Anupam ChandMetadata driven framework for Delta Live tablesDelta live tables was made generally available around 2022. So it fairly new in the industry and everyone is trying to get an understanding…May 11May 11
Anupam ChandPyspark data frame quality validation framework in Databricks using Great Expectations(hands on)We all know how important data quality for a data platform and data analysis. Databricks in one of popular platforms used to conduct ETL…Dec 21, 20232Dec 21, 20232
Anupam ChandFree online services for a Dev/Sftware EngineerI wrote this article as a reference for free online services which can be used either for quick POCs or to learn a new technology. If you…Dec 14, 2023Dec 14, 2023
Anupam ChandTrigger a Databricks workflow from Azure Data factory using Managed IdentityThis article assumes that you already know about Databricks workflows. We are going to see how we can securely trigger this using Data…Dec 14, 2023Dec 14, 2023
Anupam ChandCosmosDB client singleton using PythonWhenever you are working with CosmosDB, you will find that the online documentation recommends that you use a singleton client instance for…Nov 22, 20231Nov 22, 20231
Anupam ChandTrigger Databricks job via RestApi using oAuth(AAD)So I had written an article earlier on how to trigger a databricks spark job via rest api (Link). However, that article demonstrates the…Aug 1, 20232Aug 1, 20232
Anupam ChandSave Databricks costs by using Spot instances with job clustersFrom my earlier article, we saw the benefits of using a job cluster over an interactive cluster for databricks batch jobs. Today we are…Jul 14, 2023Jul 14, 2023
Anupam ChandAzure Databricks job clusters — Installing python and maven libariesWhen triggering Databricks notebooks from datafactory, there is a choice of choosing an interactive cluster or a job cluster. Job cluster …May 23, 2023May 23, 2023
Anupam ChandGeneration of sFTP keys and load to Keyvault for ADF or Logic appsThere are 2 methods which will be covered in this page to generate SSH keys.Aug 29, 20221Aug 29, 20221
Anupam ChandPGP encryption using python in Azure databricksMany data analytics platforms ingests files in the order of terabytes. Some of these files contain sensitive data and the organization may…Aug 16, 20221Aug 16, 20221
