Explore and reuse SAP Datasphere models from Databricks

[remi_astier]

As more organizations build on the SAP Business Data Cloud (BDC), a recurring challenge emerges: how do you bring Databricks workloads and ML pipelines closer to the semantic models that already live in SAP Datasphere — without duplicating the logic?

Datasphere's semantic layer is where analytical value is concentrated. Whether surfaced as Analytical Models — with explicitly defined measures, dimensions, and hierarchies — or as structured views that encapsulate complex join logic, these artifacts are the foundation for interactive stories in SAP Analytics Cloud.

When Databricks teams want to consume this data, two paths exist:

• Rebuild: Export raw datasets as data products, share them with Databricks, then manually recreate all joins, measures, and hierarchies in the Databricks layer.
• Consume the models directly: Access data through a database connection that reads from the semantic layer itself — preserving the existing logic, as shown in the diagram below.

The second approach is the focus of this post, displayed in red in the below diagram.

Direct connections from Databricks to SAP Datasphere are useful beyond read access — common patterns include writing prediction results back to Datasphere, or running exploratory work before committing to a full data product.

Getting a basic connection working is straightforward — production-ready security is not. Two requirements raise the bar significantly:

• Certificate-based authentication, replacing the simpler but weaker password approach
• Databricks Secrets, to store, rotate, and audit credentials in a controlled way

Meeting both typically means piecing together several documentation pages and adapting commands from obsolete tutorials. This post consolidates everything into a single, end-to-end walkthrough.

Trusting a certificate requires a certificate authority

The first step is registering a trusted certificate authority (CA) in Datasphere. The concept is easiest to grasp through an analogy: a border officer can't verify every traveler's documents from scratch — that would be unworkable at scale. Instead, they trust Spain's passport authority, so any document bearing its official stamp is accepted without question. Certificate authorities work on exactly the same principle.

In a production environment, your company's security team plays that role: you submit a certificate signing request, and they return a signed certificate that Datasphere will trust. Before diving into those steps, we'll build a self-signed CA from scratch — both to make the setup self-contained and to make the underlying mechanics concrete.

To demonstrate the x509 authentication protocol, we'll generate two certificates: a CA certificate that acts as the trusted root, and a user certificate signed by that CA, which Datasphere will use to verify the identity of the connecting client.

Create a certificate authority

The commands below generate a public/private key pair to act as our certificate authority — the only prerequisites are a Linux shell and openssl.

CANAME=FakeCompany
O="FakeCompany SE"
OU="FakeCompany IT Security Team"
openssl req -nodes -new -x509 -keyout $CANAME.key -out $CANAME.crt -days 360 \
  -subj "/CN=$CANAME/C=FR/ST=PA/L=Paris/O=$O/OU=$OU"

That command generated the public/private key pair for our certificate authority. In a real organization, the private key would be among the most sensitive digital assets in existence — typically stored in dedicated hardware (an HSM) or on an air-gapped machine locked in a secure room. For the purposes of this walkthrough, we'll keep it as a file in the current directory.

The last parameter of the command sets a few attributes: Common Name (CN), Country (C), state (ST), Locality (L), Organization (O), Organization Unit (OU)

Create a client certificate

Certificates can be issued by other departments of the organization, here we pick “Operational Data Science Team” and a common name “X509 IS FUN”.

MYCERT=X509ISFUN
openssl req -new -nodes -out $MYCERT.csr -newkey rsa:4096 -keyout $MYCERT.key \
  -subj "/CN=${MYCERT}/C=FR/ST=PA/L=Paris/O=$O/OU=Operational Data Science Team"

The file with the extension .csr is the signing request, which is signed by the IT security team that has access to the private key of the certificate authority.

openssl x509 -req -in $MYCERT.csr -CA $CANAME.crt -CAkey $CANAME.key -CAcreateserial -out $MYCERT.crt -days 60

For authentication, we’ll later need to bundle the client key pairs and the authority public key into a PEM file or a PSE.

cat $MYCERT.key $MYCERT.crt $CANAME.crt > $MYCERT.pem

Register the certificate authority

In Datasphere open System -> Configuration, navigate to the tab “Security”

Click on the + sign to upload the public key of the dummy certificate authority.

After selecting the file, choose x509  for “purpose”

Create a user for the client certificate

Next navigate to space management, pick a space with the models you're interested in and open the tab “database access”.

Choose a username and certificate based authentication and the signed client public key file.

As you can see above, attributes of the certificate and the signature are displayed.

Tick the boxes for APL+PAL , read access to the space schema, and create permission on the personal schema to make temporary tables.

Test the connection

Test authentication with the command line tool hdbsql from the HANA client.

hdbsql -n aa034565-**************.hna2.prod-eu10.hanacloud.ondemand.com:443 \
 -u "GCOE_DSP#$MYCERT" -e -sslprovider openssl \
 -Z authenticationMethods=x509 \
 -Z authenticationX509=./$MYCERT.pem \
 -j "select * from DUMMY"

DUMMY
"X"
1 row selected (overall time 972.140 msec; server time 282 usec)

If the test fails with a network error, the datasphere instance firewall might block connections from databricks. If that is the case, check the appendix 2.

Store certificate and connection details in (SAP) Databricks

Using the Databricks command line tool, log into your workspace url.

databricks auth login https://dbc-a2***.cloud.databricks.com/

Scopes are containers of sensitive data such as credentials. Their use is audited, and the content can be used in code without being exposed in plain text.

databricks secrets create-scope demo-$CANAME

Now we put the address of Datasphere and the user name into a json file

echo "{
\"address\": \"aa034565-************.hna2.prod-eu10.hanacloud.ondemand.com\",
\"port\": 443,
\"user\": \"GCOE_DSP#$MYCERT\",
\"authenticationMethods\": \"X509\",
\"authenticationX509\": \"certificate is in another key\" }" > DSP-$MYCERT-CONNECTION.json

Upload the two files

cat DSP-$MYCERT-CONNECTION.json | databricks secrets put-secret demo-$CANAME DSP-$MYCERT-CONNECTION

cat $MYCERT.pem | databricks secrets put-secret demo-$CANAME DSP-$MYCERT-PEM

Explore Datasphere models from a notebook in databricks

Paste the following python commands in notebook cells.

!pip install hana-ml

from json import loads as json_loads
MYCERT="X509ISFUN"
CO="FakeCompany"
con_details=json_loads(dbutils.secrets.get(scope=f"demo-{CO}", key=f"DSP-{MYCERT}-CONNECTION"))

con_details.update({
    "authenticationX509": dbutils.secrets.get(scope=f"demo-{CO}", key="DSP-X509ISFUN-PEM")
})

from hana_ml import ConnectionContext
con=ConnectionContext(**con_details) 
con.is_cloud_version()

 Hopefully you should be connected from your notebook environnement to Datasphere now 😀🎉🍾

Explore a view

Let’s explore the view V_I_GoodsMovement, in the space GCOE_DSP  using a notebook.

df1=con.table("V_I_GoodsMovement", schema='GCOE_DSP')

#For simplicity, let’s look only at the first 30 columns
df2=df1.select(df1.columns[:30])

We can use the hana-ml python library to generate a profile of the dataset

from hana_ml.visualizers.dataset_report import DatasetReportBuilder
drb=DatasetReportBuilder()
drb.build(df2)
drb.generate_notebook_iframe_report()

And this concludes the main topic which was setting up a secure connection from Databricks to the HANA database of SAP Datasphere. 👍

Appendix 1.  Query the audit trail in Databricks to review the use of 

SELECT event_time, user_identity.email AS user_email, action_name, request_params.scope AS secret_scope, request_params.key AS secret_key, source_ip_address, user_agent
FROM system.access.audit
WHERE action_name = 'getSecret'
 and request_params.scope='demo-FakeCompany'
ORDER BY event_time DESC LIMIT 10

Appendix 2: whitelist SAP Databricks (serverless) addresses in datasphere firewall

This python snippet to be run in a notebook in the databricks instance should display the subnet addresses to add to the firewall in datasphere. It looks up the current outbound gateway, then find in the databricks network reference, which other gateways can be used with load balancing.

from ipaddress import IPv4Address, IPv4Network
from requests import get as httpget

r=httpget('https://www.ifconfig.me/ip')
i=IPv4Address(r.text)
print(f"Current outbound gateway IP address is {i}")

# there are other possible outgoing IP addresses, let's lookup the other possible ones
r=httpget('https://www.databricks.com/networking/v1/ip-ranges.json')
outbound_subnets= [ subs for subs in r.json()['prefixes'] if subs['type']=='outbound' ]

whitelist=[]

for s in outbound_subnets:
    for p in s['ipv4Prefixes']:
        if i in IPv4Network(p):
            whitelist=s['ipv4Prefixes']
            break

if len(whitelist)>0:
    print(f"To allow network traffic initiated from this databricks instance, you must whitelist the following subnets: {whitelist}")
else:
    print("Not sure what is happening here. ")

 

And then open System / Configuration and navigate to the tab "IP Allowlist" to add the entries