S/4Hana to Kafka

[werner_daehn]

More and more customers use Apache Kafka as their real-time data backbone. SAP Hana contains the current state of the business and Apache Kafka the entire stream of changes since the beginning. This enables all data consumers to get ERP data without accessing the expensive S/4Hana system, thus is a great cost saving measure and open new possibilities.

Because Kafka is so poplar, most tools support Kafka, e.g. SAP Data Intelligence, all Big Data tools, ETL tools, pretty much everything.

What customers are missing is an easy way to get S/4Hana data into Kafka, though and the S/4HanaConnector for Kafka helps here (see github and docker).

The usage of the S/4HanaConnector is very simple:

1. Pull it from Docker Hub


2. Open the Admin UI and create connections to the S/4Hana system and Kafka


3. Select the objects to produce data for


4. Assign the objects to one or multiple producer instances

Everything else happens under the cover. The Kafka schema definitions are derived from the SAP structures, at first start an initial load is performed and from then on the data is produced with a latency of seconds.

With this connector it is a matter of minutes to get data into Kafka.

 

Under the covers there is much more going on, everything needed for a complete Data-, System- and Process Integration solution. Metadata about the landscape, impact/lineage information, ability to map data when reading - e.g. to rename the columns - and to adjust it to an existing schema instead of using a source specific schema.

This post is part of a series and its full power is unlocked when combining it with one of the other components.