Connecting SAP Analytics Cloud to Databricks model serving endpoint

[Ian_Henry]

Scenario

Imagine your data scientists have worked their magic and built a predictive (I should call it AI now) model that you wish to access from an SAP Analytics Cloud (SAC) story. The model may require parameters to analyze specific data, or you may require the latest output to be retrieved

This can be achieved using the SAC Multi Action feature. Databricks and SAC both support OAuth 2.0 for authentication. A service principal is used for authorization and authentication outside of the standard Databricks UI. The SAC Multi Action feature offers different actions(steps), including an API step which utilises a secure HTTP API connection

Prerequisites

Required Solutions

SAP Analytics Cloud
Databricks

Required users access

SAP Analytics Cloud BI Admin or SAP Analytic Cloud Planning
Databricks Workspace Admin

Required artefacts

Databricks Model Serving Endpoint

OAuth with Databricks / SAP Databricks

The steps are the same with native Databricks and SAP Databricks. The Databricks documentation describes the generic steps and required parameters for OAuth 2.0 authorization. 
https://docs.databricks.com/aws/en/dev-tools/auth/oauth-u2m 
The SAC HTTP API connection requires 5 parameters from Databricks.

• Data Service URL
• OAuth Client ID
• Secret
• Token URL
• Scope

Databricks Service Principal

We must first create service principal in Databricks, this can be at either the account or the workspace level. 
Here I have created the service principal within the workspace.

Navigate to User Settings -> Identity and Access

Identity and Access

Add Service Principal

Add Service Principal

Add Service Principal Continued

Add Service Principal Name

OAuth Secret

With the Service Principal created we can generate the secret required for authentication

Service Principal

Service Principal Secrets

Service Principal Generate OAuth Secret

Generate OAuth Secret - Lifetime

We need the secret and client details for the SAC connection, copy them somewhere safe

Generate Secret

Serving Endpoint Permissions

The service principal requires execute/query permission on the model serving end point.
We can add the permission via code or the UI as below

Model Serving Endpoint

Model Serving Endpoint Details

Model Serving Endpoint Permissions

SAP Analytics Cloud Connection

Switch to SAC, we need to define a new HTTP API connection

SAP Analytics Cloud HTTP API ConnectionSAP Analytics Cloud HTTP API Connection DialogueSAP Analytics Cloud HTTP API Connection with Placeholders

When we click "Create" it validates the connection details.

SAC Multi Action

Define a new SAC Multi Action using the connection defined.

SAP Analytics Cloud New Multi ActionMulti Action - Add API StepMulti Action API Step

Within the Multi Action properties we can select the previously defined connection. 
The API Request URL is required again here, we can paste in the Model Serving Endpoint here.

SAP Analytics Cloud Story

The multi action defined above needs to be included within an SAC story

SAC Story Add Multi Action Starter ComponentSAC Multi Action Starter Properties

We can test the Multi Action call by clicking on the play button. 

SAC Execute Multi ActionSAC Multi Action Complete

All being well, once completed we can see an additional notification available

SAC Multi Action Notifications

SAP Analytics Cloud Job Monitor

The Multi Action history is captured within the Job Monitor

SAP Analytics Cloud - System Job MonitorJob Monitor History

Conclusion

In this blogpost I have shown that integrating a Databricks model serving endpoint can be achieved with only a few steps using out of the box features from Databricks and SAP Analytics Cloud.