Anupam ChandMetadata driven framework for Delta Live tablesDelta live tables was made generally available around 2022. So it fairly new in the industry and everyone is trying to get an understanding…10 min read·May 1, 2024----
Anupam ChandPyspark data frame quality validation framework in Databricks using Great Expectations(hands on)We all know how important data quality for a data platform and data analysis. Databricks in one of popular platforms used to conduct ETL…7 min read·Dec 21, 2023--2--2
Anupam ChandFree online services for a Dev/Sftware EngineerI wrote this article as a reference for free online services which can be used either for quick POCs or to learn a new technology. If you…2 min read·Dec 14, 2023----
Anupam ChandTrigger a Databricks workflow from Azure Data factory using Managed IdentityThis article assumes that you already know about Databricks workflows. We are going to see how we can securely trigger this using Data…3 min read·Dec 14, 2023----
Anupam ChandCosmosDB client singleton using PythonWhenever you are working with CosmosDB, you will find that the online documentation recommends that you use a singleton client instance for…5 min read·Nov 22, 2023--1--1
Anupam ChandTrigger Databricks job via RestApi using oAuth(AAD)So I had written an article earlier on how to trigger a databricks spark job via rest api (Link). However, that article demonstrates the…2 min read·Aug 1, 2023--1--1
Anupam ChandSave Databricks costs by using Spot instances with job clustersFrom my earlier article, we saw the benefits of using a job cluster over an interactive cluster for databricks batch jobs. Today we are…4 min read·Jul 14, 2023----
Anupam ChandAzure Databricks job clusters — Installing python and maven libariesWhen triggering Databricks notebooks from datafactory, there is a choice of choosing an interactive cluster or a job cluster. Job cluster …5 min read·May 23, 2023----
Anupam ChandGeneration of sFTP keys and load to Keyvault for ADF or Logic appsThere are 2 methods which will be covered in this page to generate SSH keys.5 min read·Aug 29, 2022----
Anupam ChandPGP encryption using python in Azure databricksMany data analytics platforms ingests files in the order of terabytes. Some of these files contain sensitive data and the organization may…5 min read·Aug 16, 2022----
