Sharing Data Products from SAP Databricks to SAP B...

Yogananda

This blog explains how to use Python SDK for publishing data products in SAP Business Data Cloud so that it becomes discoverable by other consumer applications in SAP Business Data Cloud, for example by SAP Datasphere.

Architecture at High level

Prerequisites - Adding Privileges into SAP Databricks

Steps to follow (Screenshots)

Access your SAP Databricks Workspace environment. From the sidebar menu, choose Catalog.
Choose the Manage (Gear icon) button and, from the dropdown menu, select Metastore.
Choose the Permissions tab and then choose the Grant button.
In the Principals field, enter the user or a group of users.
Select the boxes CREATE CATALOG, CREATE SHARE, SET SHARE PERMISSION, USE PROVIDER, USE RECIPIENT, and USE SHARE then choose Confirm.
You have added the necessary privileges.

For a Delta Share to be exposed for downstream processing in SAP Databricks, it needs to be published as a data product with rich metadata. This can be done using an iPython Notebook available in your SAP Databricks Workspace. For further details on how to describe the data product, see Appendix A.

Steps to follow (Screenshots)

Access your SAP Databricks Workspace environment.
From the sidebar menu, choose Catalog.
Choose the Manage (Gear icon) button and, from the dropdown menu, select Delta sharing.
Under Share data, provide the following information:
1. Create share: In the Share name field, enter a share name and choose Save and continue. The Description field is optional.
2. Add data assets: Select one or more schemas, which includes all the tables in the schema, or one or more individual tables. Choose Save and continue.
3. Add notebooks: (Optional) Specify the Storage location. Choose Save and continue.
4. Add recipients: In the Recipient field, from the dropdown menu, select sap-business-data-cloud. Choose Share data.
The Delta Share is now exposed to the workspace.

Create a folder in the workspace by choosing Workspace Create Folder. Enter a name for this new folder and choose Create.
Create a notebook, in the newly created folder, by selecting that folder and choosing Create Notebook.
Follow this steps from Learning Tutorial - https://learning.sap.com/courses/bt_bdc/dbx-enhance-data-product
Update the serverless environment version by following these steps:
1. On the sidebar menu, choose the Environment button.
2. In the Environment version field, use the dropdown menu to choose 3.
3. Choose Apply.
Hover over the area under the rectangle and use +Code to create a cell in the notebook to run the instructions.
In this cell, install sap-bdc-connect-sdk:

Installation

To install this SDK use the following pip command:

pip install sap-bdc-connect-sdk

uv tool install sap-bdc-connect-sdk

Python Documentation for BDC Connect - https://pypi.org/project/sap-bdc-connect-sdk/

Create or update the share in sap-bdc-connect-sdk.

A share is a mechanism for distributing and accessing data across different systems. Creating or updating a share involves including specific attributes, such as @openResourceDiscoveryV1, in the request body, aligning with the Open Resource Discovery protocol. This procedure ensures that the share is properly structured and described according to specified standards, facilitating effective data sharing and management.

Copy the below code snippet and paste it to the cell, then provide the share-name, title, shortDescription and description of your share.

from bdc_connect_sdk.auth import BdcConnectClient
from bdc_connect_sdk.auth import DatabricksClient

bdc_connect_client = BdcConnectClient(DatabricksClient(dbutils, "<recipient-name>"))

share_name = "<share-name>"

open_resource_discovery_information = {
    "@openResourceDiscoveryV1": {
        "title": "<title>",
        "shortDescription": "<short-description>",
        "description": "<description>"
    }
}

bdc_connect_client.create_or_update_share(
    share_name,
    open_resource_discovery_information
)

Create or update share CSN.

from bdc_connect_sdk.auth import BdcConnectClient
from bdc_connect_sdk.auth import DatabricksClient
from bdc_connect_sdk.utils import csn_generator

bdc_connect_client = BdcConnectClient(DatabricksClient(dbutils, "<recipient-name>"))

share_name = "<share-name>"

csn_schema = csn_generator.generate_csn_template(share_name)

bdc_connect_client.create_or_update_share_csn(
    share_name,
    csn_schema
)

If the generated CSN needs to be modified for any reason, it can be written to a file for editing, for example.

			
from bdc_connect_sdk.utils import csn_generator
import json

share_name = "<share-name>"
csn_schema = csn_generator.generate_csn_template(share_name)

file_path = f"csn_{share_name}.json"
with open(file_path, "w") as csn_file:
    csn_file.write(json.dumps(csn_schema, indent=2))

# change the file

with open(file_path, "r") as csn_file:
    changed_csn_schema = csn_file.read()

csn_schema = changed_csn_schema

Publish the data product.

from bdc_connect_sdk.auth import BdcConnectClient
from bdc_connect_sdk.auth import DatabricksClient

bdc_connect_client = BdcConnectClient(DatabricksClient(dbutils, "<recipient-name>"))

share_name = "<share-name>"

bdc_connect_client.publish_data_product(
    share_name
)

Unpublishing a Data Product

from bdc_connect_sdk.auth import BdcConnectClient
from bdc_connect_sdk.auth import DatabricksClient

bdc_connect_client = BdcConnectClient(DatabricksClient(dbutils, "<recipient-name>"))

share_name = "<share-name>"

bdc_connect_client.delete_share(
    share_name
)

Follow me for more updates - https://profile.sap.com/u/Yogananda