SAP Data Intelligence offers a powerful feature that allows users to create custom operators using Python. This blog post demonstrates how to access cloud connector resources directly from within the custom Python operator. By combining the Cloud Connector with SAP Data Intelligence Python Operators, users can effortlessly connect to an even wider range of on-premise systems that may not be supported by standard operators.
Access HTTP-based resources:
Scenario: We aim to establish a connection with an HTTP-based API, such as a RESTful API or an OData service, using the Python operator in SAP Data Intelligence. To achieve this, we will configure the cloud connector, which involves a series of steps. For the purposes of this demonstration, we will use an example of a "Hello World" server that is running on my local computer. The ultimate objective is to access this server from within SAP Data Intelligence using the Cloud Connector.
Before proceeding with the steps outlined in this guide, it is essential to have an instance of the SAP Cloud Connector installed. While it is possible to install the cloud connector on a server, for the purposes of this demonstration, we will be using a Windows machine. We recommend following the instructions provided in this blog (https://blogs.sap.com/2021/09/05/installation-and-configuration-of-sap-cloud-connector/) to install and configure the cloud connector.
Second requirement is a BTP subaccount with a Data Intelligence cluster. To this subaccount we will connect the Cloud Connector.
The first step is to create a configuration in the Cloud Connector that connects to our subaccount and exposes the HTTP resource. For the purpose of this demonstration, I ran a small Node.js server on my Windows machine that outputs "Hello World".
Localhost Example Server
To create the configuration, navigate to the admin interface for the cloud connector and create a "Cloud to On-Premise" configuration.
Cloud To On-Premise Configuration
In the screenshot above, you can see that I exposed the internal host "localhost" with port 3333 via a virtual host called "virtualhost". This virtual host is the host that we will be requesting from the BTP side. For the time being, I exposed all paths using an unrestricted access policy, but in production scenarios, access policies can be defined more granularly.
BTP Registered Cloud Connectors
On the BTP end, we can check the cockpit and the connected cloud connectors in the respective menu tab. If you cannot see this tab, you may be missing some roles. It is important to note that we see the LocationID "FELIXLAPTOP", which is an identifier that distinguishes multiple cloud connectors connected to the same subaccount.
2. Creating a Data Intelligence Connection:
For our purposes we do not want to hard-code the connection details, because we need a little help from the connection management to access the Connectivity Service of BTP. In the Connection Management Application from SAP Data Intelligence we can create connections of various types. In this case, we create a connection of type HTTP with host, port and SAP Cloud Connector as the gateway.
res = requests.get(url=url, headers=headers, proxies=proxies)
The Python script is sending a request using Python's standard requests library. We use the connection details that were created and stored by the Connection Management. To specify the Cloud Connector as a proxy, we provide a host, port, and the proxy-authentication header.
Notably, the bearer token required to access the connectivity proxy is generated by Data Intelligence itself. To specify which LocationId to route the request to, we specify the "SAP-Connectivity-SCC-Location_ID" header.
The created connection can be accessed using api.config.http_connection, and the logs reveal the various fields that are accessible.
Interestingly we can see that the connectivity service host is resolved locally in Data Intelligence. It is a internal proxy to the BTP connectivity service that handles the connectivity to the Cloud Connector.
4. Testing the Custom Operator:
Finally we can create a new graph, drag the operator in and fill the http_connection parameter. In this instance I provide the ID of the connection I created previously.
HTTP Connection Parameter
When executing the graph, we can have a look at the console and see the following entries:
That means we were successful and the operator was able to request the local HTTP server on my Windows machine.