Overview:
This blog demonstrates how to leverage SAP Business Data Catalog (BDC) in conjunction with Enterprise Databricks to create custom data products, extend them with advanced analytics and transformations, and synchronize the refined data back to SAP BDC. This integrated approach enables organizations to maximize data value while maintaining governance and traceability across their data infrastructure.
Main steps at a glance
1. Check the prerequisites
a. Enable your SAP BDC Core in SAP4Me portal, it important to provision it in the same hyperscaler and region of your Enterprise Databricks or in the future your GCP or Microsoft Fabric tenants, because you may incur some egress fees when you share data products through a BDC Connect instance. You can refer to this blog Introductory Guide to Provisioning of SAP Business Data Cloud
b. Provision your SAP Datasphere tenat in SAP BDC through SAP4Me you can refer to this blog Provisioning of Business Data Cloud : SAP Datasphere when you provision at least allocate 4,150 CU because you need to activate object store in you tenant, so you need to increase the storage to 384 GB and memory to 128 GB and in the object store you need to add 1 block for storage, 1 block for memory and 1 block for requests you can use the SAP Business Data Cloud Capacity Unit Estimator to configure your business case
c. Create one HANA Data Lake Files space in you Datasphere tenant, and add it to an scope role to provide access
d. Provision your SAP BDC Connect instance in SAP4Me portal, you can check this blog How to provision SAP BDC Connect for Databricks?
e. Prepare your source system with all the configuration needed to connect with you SAP Datasphere tenant, in this document you can check all the connection types and prerequisites -> SAP Datasphere Connection Types
f. If you want to connect an S4HANA system then you need to configure your cloud connector for S4HANA PCE or S4HANA On-Premise connection with SAP Datasphere:
- Step 1: Setup SAP Cloud Connector connect to SAPDatasphere
- Step 2: Setup mapping (RFC connection) in Cloud Connector for a SAP S/4HANA On-Premise connection
- Step 3: Create SAP S/4HANA On-Premise Connection for Data Flow and Replication Flow
- Step 4: Replication Flow for CDS views from S/4HANA On-Premise to SAP Datasphere
2. Create the formation
a. Create your formation in SAP4Me portal. Go to Portfolio & Products -> My Product Packages tab-> Business Data Cloud product package -> Customer Landscape tab -> Formations sub-tab and choose Create Formation, enter a unique formation name and choose Integration with SAP Business Data Cloud
b. Include the following mandatory systems: SAP Business Data Cloud, SAP Datasphere and SAP BDC Connect instance.
3. Create custom managed data product
a. Open your SAP Datasphere tenant. First of all replicate your data. You can replicate directly to object store creating a connection from your source system to SAP Datasphere in the HANA Data Lake Files space or you can replicate an existing table coming from another space in your Datasphere tenant. If you want to use a view or analytic model you must materialize the view or model in a table, then replicate into a table in your HANA Data Lake Files space. My table is TL_ADW_OBSESSION
b. Create a formation profile in your SAP Datasphere tenant. Go to Data Sharing Cockpit -> My Data Provider Profile -> Create Data Provider Profile. In this example my data provider profile is ONB_Data_Products
c. Go to Data Sharing Cockpit -> My Data Products -> Click on Create Product (+ icon)
d. In Artifact Space select the object store space for your data products
e. Select your table or tables, in my case is TL_ADW_OBSESSION and click on Save Changes
f. Click on Switch Status and select List. When the data product is Listed then we can share it with Databricks
g. Open your SAP Business Data Cloud Cockpit and go to Catalog & Marketplace -> Search and we look for Demo Onboarding data product, open it and click on Add to Cart
h. Select SAP BDC Connect for Databricks and click on Add to Cart
i. Click view request cart and click on Proceed
j. Provide the details for the purpose and access period, click on the check box to accept the terms and conditions and click on Send
k. Now the data product administrator need to approve the access request. Go to Governance -> Access Management, select the request and in request details section click on Approve
l. Finally go to Catalog & Marketplace -> Data Product Access -> Agreements select your data product and in the agreements details click on Install Data Product. When the installation status change to Installed now you can go to Databricks
4. Extend and push it back to SAP BDC
a. Open your Enterprise Databricks workspace and go to Catalog -> At the top of the Catalog pane, click the gear icon and select OpenSharing
b. Click on your BDC Connect provider
c. Click on Mount to Catalog
d. Create a new catalog or select an existing one. I selected an existing one and then click on Mount
e. Go back to your Catalog and in Shares received you can see bdc_deltashare catalog with the table coming from Datasphere TL_ADW_OBSESSION
f. Extend the table with advanced analytics and/or transformations. Then new table in Databricks is cluster_demo, the next step is create a share. Go to Catalog -> At the top of the Catalog pane, click the gear icon and select OpenSharing. Click on Shared by me tab and click Share data
g. Create a share, enter the share Name and add an optional comment. Click on Save and continue
h. On the Add data assets section, select the data assets you want to share. In this case I want to share the table cluster_demo. Then click on Save and continue
i. In the next step select the notebooks. Click on Save and continue
j. In the final step select your BDC Connect provider as recipient. Then click on Share data
k. Now we have a new share with the name SAP_BDC_Connect. With SAP_BDC_Connect we are going to share the table cluster_demo
l. Then please go to Catalog -> At the top of the Catalog pane, click the gear icon and select OpenSharing. Click on Shared by me tab, click Recipients and select the recipient
m. Click on Grant share. On the Grant share dialog, start typing the share name or click the drop-down menu to select the shares you want to grant, finally click on Grant
n. Now you need to publishing the share as a Data Product to SAP Business Data Cloud. The next image is an example of the code, for more details please review the following documentation -> Databricks documentation: Grant SAP Business Data Cloud (BDC) recipients access to OpenSharing data ...
o. Then we come back to SAP BDC Cockpit and we should be able to see our new data product coming from Enterprise Databricks
Summary:
In this blog, we explore how to create a custom data product in SAP Datasphere, the steps to share it with an SAP BDC Connect instance, extend it in Databrick by creating a new cluster table and finally, send it back to SAP Business Data Cloud as a new data product.
Bluesky
