SAP Data Intelligence Python Operators and Cloud Connector - HTTP

[felixbartler]

SAP Data Intelligence offers a powerful feature that allows users to create custom operators using Python. This blog post demonstrates how to access cloud connector resources directly from within the custom Python operator. By combining the Cloud Connector with SAP Data Intelligence Python Operators, users can effortlessly connect to an even wider range of on-premise systems that may not be supported by standard operators.

Access HTTP-based resources:

Scenario: We aim to establish a connection with an HTTP-based API, such as a RESTful API or an OData service, using the Python operator in SAP Data Intelligence. To achieve this, we will configure the cloud connector, which involves a series of steps. For the purposes of this demonstration, we will use an example of a "Hello World" server that is running on my local computer. The ultimate objective is to access this server from within SAP Data Intelligence using the Cloud Connector.

Szenario Architecture

Prerequisite:

Before proceeding with the steps outlined in this guide, it is essential to have an instance of the SAP Cloud Connector installed. While it is possible to install the cloud connector on a server, for the purposes of this demonstration, we will be using a Windows machine. We recommend following the instructions provided in this blog (https://blogs.sap.com/2021/09/05/installation-and-configuration-of-sap-cloud-connector/) to install and configure the cloud connector.

Second requirement is a BTP subaccount with a Data Intelligence cluster. To this subaccount we will connect the Cloud Connector.

1. Configuration:

The first step is to create a configuration in the Cloud Connector that connects to our subaccount and exposes the HTTP resource. For the purpose of this demonstration, I ran a small Node.js server on my Windows machine that outputs "Hello World".

Localhost Example Server

To create the configuration, navigate to the admin interface for the cloud connector and create a "Cloud to On-Premise" configuration.

Cloud To On-Premise Configuration

In the screenshot above, you can see that I exposed the internal host "localhost" with port 3333 via a virtual host called "virtualhost". This virtual host is the host that we will be requesting from the BTP side. For the time being, I exposed all paths using an unrestricted access policy, but in production scenarios, access policies can be defined more granularly.

BTP Registered Cloud Connectors

On the BTP end, we can check the cockpit and the connected cloud connectors in the respective menu tab. If you cannot see this tab, you may be missing some roles. It is important to note that we see the LocationID "FELIXLAPTOP", which is an identifier that distinguishes multiple cloud connectors connected to the same subaccount.

2. Creating a Data Intelligence Connection:

For our purposes we do not want to hard-code the connection details, because we need a little help from the connection management to access the Connectivity Service of BTP. In the Connection Management Application from SAP Data Intelligence we can create connections of various types. In this case, we create a connection of type HTTP with host, port and SAP Cloud Connector as the gateway.

Note: Not all connection types allow you to access via the Cloud Connector. See the official product documentation for details.

3. Developing a Custom Operator:

In the Operators menu of Data Intelligence we create a new Operator based on the Python3 Operator.

Python Operator Creation

We than change the configSchema.json of this operator to accept a HTTP Connection as parameter. This file can be found in the repository under the following path.

vflow\subengines\com\sap\python36\operators\PythonCloudConnector\configSchema.json

The JSON file looks like this:

{

    "$schema": "http://json-schema.org/draft-06/schema#",

    "$id": "http://sap.com/vflow/PythonCloudConnector.configSchema.json",

    "type": "object",

    "properties": {

        "codelanguage": {

            "type": "string"

        },

        "scriptReference": {

            "type": "string"

        },

        "script": {

            "type": "string"

        },

        "http_connection": {

            "title": "HTTP Connection",

            "description": "HTTP Connection",

            "type": "object",

            "properties": {

                "configurationType": {

                    "title": "Configuration Type",

                    "description": "Configuration Type",

                    "type": "string",

                    "enum": [

                        " ",

                        "Configuration Manager",

                        "Manual"

                    ]

                },

                "connectionID": {

                    "title": "Connection ID",

                    "type": "string",

                    "format": "com.sap.dh.connection.id"

                },

                "connectionProperties": {

                    "title": "Connection Properties",

                    "description": "Connection Properties",

                    "$ref": "http://sap.com/vflow/com.sap.dh.connections.http.schema.json",

                    "sap_vflow_constraints": {

                        "ui_visibility": [

                            {

                                "name": "configurationType",

                                "value": "Manual"

                            }

                        ]

                    }

                }

            }

        }

    },

    "required": [

        "http_connection"

    ]

}

And finally to the interesting part, the Python implementation of the actual HTTP request:

import requests



http_connection = api.config.http_connection



api.logger.info(str(http_connection))



def gen():

    api.logger.info("Generator Start")

    

    url = f"""http://{http_connection["connectionProperties"]["host"]}:{http_connection["connectionProperties"]["port"]}"""



    headers = {

        "proxy-authorization": "Bearer " + http_connection["gateway"]["authentication"],

        "SAP-Connectivity-SCC-Location_ID": "FELIXLAPTOP"

    }



    proxies = {

        "http": f"""http://{http_connection["gateway"]["host"]}:{http_connection["gateway"]["port"]}/"""

    }

    

    res = requests.get(url=url, headers=headers, proxies=proxies)

    

    api.logger.info(res.text)

    

    api.logger.info("Generator End")



api.add_generator(gen)

The Python script is sending a request using Python's standard requests library. We use the connection details that were created and stored by the Connection Management. To specify the Cloud Connector as a proxy, we provide a host, port, and the proxy-authentication header.

Notably, the bearer token required to access the connectivity proxy is generated by Data Intelligence itself. To specify which LocationId to route the request to, we specify the "SAP-Connectivity-SCC-Location_ID" header.

The created connection can be accessed using api.config.http_connection, and the logs reveal the various fields that are accessible.

{

   "configurationType":"Configuration Manager",

   "connectionID":"HTTP_ONPREM_SYSTEM",

   "connectionProperties":{

      "authenticationType":"NoAuth",

      "host":"virtualhost",

      "port":3333,

      "protocol":"HTTP"

   },

   "connectionType":"HTTP",

   "gateway":{

      "authentication":"<token>",

      "host":"connectivity-proxy-service",

      "locationId":"",

      "password":"",

      "port":20003,

      "subaccount":"<subaccout-id>",

      "user":""

   },

   "type":"HTTP"

}

Interestingly we can see that the connectivity service host is resolved locally in Data Intelligence. It is a internal proxy to the BTP connectivity service that handles the connectivity to the Cloud Connector.

4. Testing the Custom Operator:

Finally we can create a new graph, drag the operator in and fill the http_connection parameter. In this instance I provide the ID of the connection I created previously.

HTTP Connection Parameter

When executing the graph, we can have a look at the console and see the following entries:

Generator Start

Hello World!

Generator End

That means we were successful and the operator was able to request the local HTTP server on my Windows machine.

In the next blog I will show how to create a TCP socket in Python to connect to TCP resources.

Hope you find the content of this blog helpful. Feel free to comment for further clarifications.