Automating Invoice Predictions from SAP Databricks with SAP Datasphere Task Chain

[BabacarT]

In our previous blog, we demonstrated how to process semi-structured PDF invoices, parse them and store the results in a Delta table before applying machine learning algorithms to predict payment delays. These predictions are hosted in SAP Databricks as part of a customer managed data product, with results written to a target Delta table in Unity Catalog and shared via Delta Sharing with SAP Business Data Cloud (BDC).

This is how our customer managed data product holding the first processed 1.000 invoices looks like before any updates: 

The next challenge on our journey was automation. Once new invoices land in the S3 bucket, we need a fully automated process that parses them, updates predictions, and refreshes the Delta share, without any manual intervention. Since SAP consumes the data federatively, business users expect up‑to‑date predictions in near real time.

To meet this requirement, my colleague @Mostafa_Shaarawy1 and Idesigned and implemented an end‑to‑end orchestration flow using SAP Datasphere (DSP) Task Chains and Databricks API to:

• trigger the Databricks notebook on schedule or on demand,
• monitor its execution,
• and ensure updated predictions flow instantly to SAP BDC.

To achieve this, we are going to utilize the run-now and the get run APIs described in Databricks API reference, with authentication handled via a Databricks Service Principle.

The result: a fully automated pipeline that delivers fresh, reliable insights to business users, improving decision‑making, operational efficiency, and payment‑risk visibility.

---

Business Use Case: Finance Manager’s Perspective

Imagine you’re a finance manager responsible for optimizing cash flow. You rely on accurate predictions of payment delays to decide which customers to follow up with and when. With Datasphere orchestration, the moment new invoices arrive, the system automatically processes them, updates predictions, and makes the latest insights available in SAP BDC, without any manual steps. This means you can act faster, reduce overdue payments, and improve working capital, all while saving time and effort.

We’ll achieve this using Datasphere Task Chains, leveraging their API Task (REST) capability and a Generic HTTP connection to seamlessly integrate with SAP Databricks.

Disclaimer: The scenario we are covering in this blog is technically working fine but is not officially supported in the SAP Databricks documentation.

---

Solution Overview

The architecture looks like this:

What makes this work?

• The architecture consists of three components:

  • DSP Task Chains orchestrate Databricks jobs using REST API calls.
  • Databricks Jobs API triggers and monitors the notebook execution.
  • Delta Sharing + Unity Catalog exposes the updated prediction table live to SAP BDC/DSP.

This ensures new data flows from invoice ingestion → prediction → Delta share → SAP consumption end‑to‑end, without manual steps.

---

Prerequisites

Before building the orchestration, we need to first complete these steps:

1. Create a Service Principal in SAP Databricks Workspace at the workspace level to generate:

   • Client ID
   • Client Secret
   • OAuth Token Endpoint.

2. Assign the service principal permissions to run the job and access the target Delta table.

   • This avoids using personal tokens and ensures secure automated API calls.

3. Prepare the Databricks Job

   • Identify the Job ID of the notebook that:
     • Processes new invoices,
     • Runs ML predictions,
     • Writes results into the target Delta table in Unity Catalog.

4. Update the Delta Share and the Share CSN document

---

Step-by-Step Implementation

1) Set-up Databricks and create service Principal

A service principal in Databricks is a dedicated, non-human identity built for automation. It enables secure, API‑only access to Databricks resources for scripts, automated workflows, and CI/CD pipelines, all without depending on personal user credentials.

a- Go to Settings > Ìdentity and Access > Service Principals > Add service principal:

b- Generate a secret for this Service Principal and copy it as well as the Client ID (you'll need them for the definition of the Httpconnection in

SAP Datasphere😞

c- Go to the notebook under Scheduleand create a new one:

d- Switch to the Jobs & Pipelines section and select your scheduled job and copy the job ID (you'll need it in the API Invocationrequest body of the task chain in SAP Datasphere to call the right job): 

2) Set-up your Http Connection in your Datasphere space

• In your space, navigate to Connections in Datasphere and choose Http as connection type.

• Enter the connection details.

Host   = <your_tenant_url without 'https://' >.
Port    = 443
Protocol     = HTTPS

Use Cloud Connector
    = False

Authentication Type
     = OAuth2.0

OAuth Grant Type
 = Client Credentials

OAuth Token Endpoint = <Your_token_Url_endpoint>

OAuth Scope = all-apis

OAuth Response Type: Token

OAuth Token Request Content Type: URL Encoded

Client ID: <Client_Id_of_DBX_Service_Principle>

Secret: <Secret_of_DBX_Service_Principle>

• Save the connection.

3) Create Task Chain with API Task

• Navigate to Data builder and create a new Task Chain.

• From the top of the canvas, drag and drop API Task to the editor.

From the properties of the Task Chain, select under Connection the Http connection created in step 2.

In the API Invocation section, choose POST method and the mode to be Asynchronous.

• The base url will be automatically generated from the connection details
• In the API Path, write run-now (more information can be found in the Databricks API documentation https://docs.databricks.com/api/workspace/jobs/runnow).
• The body should be a json object with the job-idof the Databricks job.

• Choose Get result from HTTP status code and response body and set the path to run_id. This will map the property run_id in the API response to be used for status retrieval to be able to monitor the Databricks job from the task chain.

In the Status API section, we will use the get single run API for status retrieval as documented in Databricks API documentation https://docs.databricks.com/api/workspace/jobs/getrun.

• Set the API path to runs/get?run_id={id}. Since we have, in the previous step, configured the path to run-id, the API task will dynamically use its value in the query parameter during run time.

• In the response section choose Get result from HTTP status code and response body to determine whether the databricks job was successful or not, based on the result from the response body.
• As success indicator use: status.termination_details.code equals to SUCCESS
• As error indicator use: status.stateequals to TERMINATED

4) Run the API task to trigger the Databricks job

Now the API task is ready to be triggered. Once triggered the Invocation API will be called with the job-id in the request body and the status API will be called periodically to evaluate the completion of the Databricks job.

For testing purpose, you can conveniently just run the API task without having to run the task chain. Click on Test Run in the API task to run it.

You can now click on the Data Integration Monitor icon to see the run details.

When you switch to the monitoring, you can then see the run details of the job:

This is also reflected in SAP Databricks:

Once the job has completed successfully, you can see the 5 invoices newly added to the S3 bucket now updated in the custom managed data product shared from SAP Databricks to SAP Business Data Cloud:

---

Key Benefits for Business Users

• Real-time insights: Data is refreshed immediately as new invoices are processed.
• Zero manual effort: A fully automated end‑to‑end pipeline.
• Improved cash flow management: Faster decisions supported by fresh insights.
• Operational efficiency: No delays between data processing and business consumption.
• Federated access: Data is consumed directly in SAP BDC—no physical copies or complex integrations required.