Trigger Databricks job via RestApi using oAuth(AAD)

So I had written an article earlier on how to trigger a databricks spark job via rest api (Link). However, that article demonstrates the calling of the api using a personal token as authentication. As mentioned in the article, this is not the safest way to do this. The aim of this article is to show how to authenticate against AAD using oAuth.

Step 1: Create an application(Service principal/SPN) on your Azure active directory tenant. You will also need to create secret to your SPN. The process to create an SPN can be easily found online in the Microsoft documentation. You can refer to this link to learn how to do this via the portal. Note down, the client id, tenant id and secret.

Step 2 : Add your new SPN to your databricks workspace. Follow this link.

Add a new SPN on your databricks workspace, Set the ApplicaionId to the client id of your SPN

Add your SPN and assign the “Can Manage Run” permissions to it.Click on “+ Add”.

Step 3 : In your databricks job, on the right side, click “Edit permissions” and add your new SPN and assign the “Can Manage Run” permissions to your SPN.

Step 4 : Fetch a token to execute your job. Fire a POST to https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/token with the following in the body(x-www-form-urlencoded).

grant_type = client_credentials

scope = 2ff814a6–3304–4ab8–85cb-cd0e6f879c1d/.default

client_id = <client_id>

client_secret = <client_secret>

Step 5 : Use the bearer token to call your databricks run job API(documented in my medium article). Just replace the personal databricks token with the received bearer token.

Hope this helps.

If there are any questions, please put them in the comments.

Don’t forget to hit that “Clap” button if this article helped you.