Triggering SAP Databricks Jobs through SAP Datasphere Task Chains

[konradlange]

Abstract

Integrating SAP Datasphere with Databricks opens up powerful possibilities for advanced analytics, machine learning pipelines, and cross-platform data processing.

In this guide, I'll share my experience automating SAP Databricks notebook execution using SAP Datasphere task chains and the Databricks Jobs API through a practical, step-by-step approach.

Common Use Cases:

• Automated data processing pipelines across SAP and Databricks
• Machine learning model training triggered by Datasphere data flows
• Advanced analytics on SAP data using Databricks' compute power
• Orchestrated cross-platform workflows with unified monitoring

Business Value:

• Reduced manual intervention in data workflows
• Faster time-to-insight with automated analytics
• Unified orchestration across enterprise data platforms

Prerequisites

Before getting started, ensure you have:

• SAP Datasphere access with permission to create connections and task chains
• Databricks workspace with notebook development access
• Databricks Personal Access Token (PAT) generation privileges
• Basic understanding of REST APIs and HTTP methods
• A Databricks notebook ready to be executed

---

Architecture Overview

The integration follows this flow:

┌─────────────────────┐
│  SAP Datasphere     │
│  Task Chain         │
└──────────┬──────────┘
           │
           │ HTTP POST Request
           │ (Jobs API)
           ▼
┌─────────────────────┐
│  Databricks         │
│  Jobs API           │
└──────────┬──────────┘
           │
           │ Triggers
           ▼
┌─────────────────────┐
│  Databricks         │
│  Notebook Run       │
└─────────────────────┘

Key Components:

1. Datasphere Task Chain - Orchestration layer
2. Generic HTTP Connection - API authentication setup
3. Databricks Jobs API - /api/2.1/jobs/runs/submit endpoint
4. PAT Authentication - Bearer token security
5. Databricks Notebook - Target execution logic

---

Step 1: Create a Generic HTTP Connection in Datasphere

The first step is to establish a secure connection from Datasphere to Databricks. We'll use a Generic HTTP Connection with Bearer token authentication.

Navigate to Connections

In SAP Datasphere, go to Connections and create a new Generic HTTP connection.

Configure Connection Settings

Setting Value Notes

Connection Name	databricks_rest_api	Choose a descriptive name
Host	<your-workspace>.cloud.databricks.com	Your Databricks workspace hostname
Port	443	Datasphere uses 443 (HTTPS) as default
Protocol	HTTPS	Always use secure connections
Path	/api/2.1 (optional)	Can be set here or in the API task later

Authentication Setup

Databricks uses Bearer tokens (Personal Access Tokens / PATs) for API authentication. For simplicity in Datasphere, we'll configure this as a Username/Password authentication type:

Setting Value

Authentication Type	Username and Password
Username	token
Password	<Your Databricks PAT>

Note: The username is literally the string token. This is how Databricks interprets Bearer token authentication via basic auth.

Obtaining a Databricks Personal Access Token (PAT)

You'll need a PAT to authenticate your API calls. Here's how to generate one:

1. In your Databricks workspace, click on your user profile in the top-right corner
2. Select Settings
3. Navigate to Developer under the User section
4. Click on Manage next to Access Tokens

1. Click Generate new token
2. Provide a description (e.g., "Datasphere Task Chain Integration")
3. Set an expiration period (recommended: 90 days for security)
4. Copy the generated token immediately as you won't be able to see it again!

Test Your Connection

Before proceeding, verify that your PAT is working correctly by making a test API call:

curl -H "Authorization: Bearer <your-PAT>" \
  https://<your-workspace>.cloud.databricks.com/api/2.1/jobs/list

If configured correctly, you should receive a JSON response listing your Databricks jobs (or an empty list if you have none as in my case).

---

Step 2: Create a Datasphere Task Chain

Now that the connection is established, let's create a task chain that will trigger the Databricks notebook.

Create a New Task Chain

1. In Datasphere, navigate to Data Builder
2. Create a new Task Chain
3. Give it a descriptive name (e.g., trigger_databricks_notebook)

Add an API Task

From the task palette, drag the API task onto the canvas.

---

Step 3: Configure the API Task

This is where we define how Datasphere will call the Databricks Jobs API.

Basic Configuration

Click on the API task to open its configuration panel.

Setting Value

Connection	databricks_rest_api (the connection you created earlier)
HTTP Method	POST

API Endpoint Configuration

The endpoint depends on how you configured your connection:

Setting Value

Full Endpoint (if Path is empty in connection)	/api/2.1/jobs/runs/submit
Partial Endpoint (if Path is /api/2.1 in connection)	/jobs/runs/submit

This endpoint is used to submit a one-off notebook run without needing a pre-configured Databricks job. For more details, see the Databricks Jobs API documentation.

Request Body

The request body defines what notebook to run and with what parameters. Here's what I used:

{
    "name": "<job-name>",
    "tasks": [{
        "task_key": "run_notebook",
        "notebook_task": {
            "notebook_path": "/Workspace/Users/<your-user>/<your-notebook>"
        },
        "environment_key": "Default",
        "timeout_seconds": 3600
    }],
    "environments": [{
        "environment_key": "Default",
        "spec": {
            "client": "1"
        }
    }],
    "max_concurrent_runs": 1
}

Request Body Breakdown

Field Purpose

name	A descriptive name for this job run (appears in Databricks UI)
tasks[].task_key	Unique identifier for this task within the job
tasks[].notebook_task.notebook_path	Full path to your Databricks notebook
tasks[].environment_key	References the environment configuration
tasks[].timeout_seconds	Maximum execution time (3600 = 1 hour)
environments[].environment_key	Environment identifier (must match task's reference)
environments[].spec.client	Client specification for the environment
max_concurrent_runs	How many instances of this job can run simultaneously

Serverless Compute Note: The example above uses Databricks Serverless Compute, so no cluster specification is required. If you're using a dedicated cluster, you'll need to add a new_cluster or existing_cluster_id field in the task definition.

Response Configuration

For monitoring purposes, configure the response handling:

Setting Value

Response Type	HTTP Status Code
Success Status Codes	200, 201

The API will return a 200 OK status if the notebook run was successfully triggered, along with a JSON response containing the run_id.

---

Step 4: Test Your Task Chain

Before deploying to production, thoroughly test your integration.

Execute the Task Chain

1. Save your task chain configuration
2. Click Run to execute the task chain manually
3. Monitor the execution in the Datasphere task chain log

Verify in Databricks

1. Navigate to your Databricks workspace
2. Go to Jobs & Pipelines
3. You should see a new run with the name datasphere_job (or whatever you specified)
4. Click on the run to see execution details and logs

Update: Migrating from Personal Access Token to OAuth 2.0

When this blog was first published, Databricks Personal Access Tokens (PATs) were the straightforward way to authenticate REST API calls from SAP Datasphere. Since then, Databricks has officially classified PATs as legacy and recommends migrating to OAuth 2.0 for all new and existing integrations. This section walks you through the updated setup.

Why the change?

Databricks has deprecated PAT-based authentication for the following reasons:

• PATs are long-lived credentials with no automatic refresh — a security risk if leaked
• New PATs are capped at a maximum lifetime of 730 days; "never expires" is no longer available
• On Azure Databricks, unused PATs are automatically revoked after 90 days
• PATs are not supported for Databricks account-level APIs at all

The recommended replacement is OAuth 2.0 Machine-to-Machine (M2M) authentication using a Databricks Service Principal with the Client Credentials grant. This gives you short-lived access tokens (~1 hour), automatic refresh, and no long-lived secrets stored in your connection.

The good news: only the HTTP connection in SAP Datasphere changes. Your Task Chain, API Tasks, and the Databricks job configuration remain exactly as described in the original blog.

Step 1: Create a Databricks Service Principal and OAuth Secret

If you do not already have a Service Principal set up for this integration, create one in Databricks:

1. In your Databricks workspace, go to Settings → Identity and access → Service principals
2. Click Add service principal and give it a descriptive name (e.g. datasphere-integration)
3. Select the newly created Service Principal and open the Secrets tab
4. Click Generate secret, set a lifetime (up to 730 days), then click Generate
5. Copy the Client ID and Client Secret immediately 

Step 2: Update the HTTP Connection in SAP Datasphere

Open the existing connection (or create a new Generic HTTP connection) in your Datasphere space and configure it as follows:

Connection Details

Field Value

Host	<your-workspace>.cloud.databricks.com
Port	443
Protocol	HTTPS
Path	/api/2.1

Authentication

Field Value

Authentication Type	OAuth 2.0
OAuth Grant Type	Client Credentials
OAuth Token Endpoint	https://<your-workspace>.cloud.databricks.com/oidc/v1/token
OAuth Scope	all-apis
Client ID	Client ID of your Databricks Service Principal
Client Secret	OAuth Secret generated in Step 1

Note: Make sure to select Client Credentials as the grant type — not Client Credentials with X.509 Client Certificate, which is a different flow.

SAP Datasphere will now automatically request a short-lived Bearer token from Databricks before each API Task execution and handle token refresh transparently. You no longer need to manage token expiry manually.

Step 3: API Tasks

The API Tasks in your Task Chain that trigger and poll the Databricks job remain unchanged:

• Run Now: POST → jobs/run-now with { "job_id": <your-job-id> } in the request body
• Poll Status: GET → runs/get?run_id={run_id} with the mapped run_id from the previous response

The only difference is that authentication now happens transparently via the OAuth connection rather than a static token header.

Summary of changes

  Before (PAT) After (OAuth 2.0)

Auth Type	User Name and Password	OAuth 2.0
Credential	Static PAT (long-lived)	Short-lived Bearer Token (auto-refreshed)
Identity	User account	Service Principal
Token expiry	Up to indefinite	~1 hour, automatic refresh
API Tasks	Unchanged	Unchanged
Databricks endpoints	Unchanged	Unchanged