SAP Datasphere & Partnerships – Confluent

[pbaumann]

Together with SAP Datasphere, SAP launched new partnerships to complete the picture of analytical data management and the idea of a Business Data Fabric.

Fig1: SAP Datasphere & Partnerships – Source: SAP, 2023 (slightly adapted)

For the other Partnerships look also here:

• SAP Datasphere & Partnerships – Collibra


• SAP Datasphere & Partnerships – Databricks


• SAP Datasphere and Partnerships – DataRobot by fazam


• SAP and Google Cloud Expand Partnership to Build the Future of Open Data and AI for Enterprises (SAP News)

I already gave an general impression of the new partners in my blog “SAP Datasphere – Q&A and Partnerships“, too. As I want to focus here on Confluent I repeat what I have written:

Confluent

Confluent, similar to Databricks, is a company build on another important open source software for data management – Apache Kafka. If you have streaming data in your company, you will not pass having a look on Kafka. Confluent delivers Kafka from the cloud as a service with an optimized ecosystem.

For data driven companies the speed of collecting and processing data in near-real-time is getting more and more important. If you search the SAP Community you will find, that Kafka is a regular topic here, too.

If you ask yourself now what exactly the difference between Confluent and Kafka is, Confluent itself gives an answer here.

So, Kafka is an important building block for an Event-driven Architecture today. I have seen customers using Kafka a Enterprise Service Bus or as backbone of their data pipelines in microservice-oriented architectures. Originally the technology was build and is still being used by LinkedIn for high throughput of streaming data.

 

Event-driven Architecture

Gartner defines Event-driven Architecture as follows:

Event-driven architecture (EDA) is a design paradigm in which a software component executes in response to receiving one or more event notifications. EDA is more loosely coupled than the client/server paradigm because the component that sends the notification doesn’t know the identity of the receiving components at the time of compiling.

In general the idea of Event-driven Architecture is not new in the SAP world. A very good overview about SAPs approach can be found in the blog SAP’s Event-Driven Ecosystem Revisited by karsten.strothmann. He also activly went into discussion about SAP Advanced Event Mesh vs Apache Kafka.

 

What already works

For sure you can find a lot of discussions about how to integrate Kafka in a SAP world like:

• Advantco KAFKA adapter for SAP HANA Smart Data Integration (SDI)


• Modern SAP ERP Data Integration using Apache Kafka


• Kafka to Hana Cloud


• S/4Hana to Kafka


• Cloud Integration – What You Need to Know About the Kafka Adapter


• Build Log-based integration with SAP Data Intelligence


• Ingesting Confluent/Kafka data into SAP Hana

In 2020 Kai Waehner, Global Field CTO of Confluent, already showed different options how to connect Kafka with SAP systems. In general he saw and discussed the following opitions:

• Traditional middleware (ETL/ESB)


• Web services (SOAP/REST)


• 3rd party turnkey solutions


• Kafka-native connectivity with Kafka Connect


• Custom glue code using SAP SDKs

 

The partnership

So why this partnership?

According to Erica Schultz, President Field Operations at Confluent

"Confluent makes it easy to connect and process real-time data streams..."

"...our customers have continuously told us that a tailored integration with SAP would significantly reduce their integration overhead, while maximizing the ROI for SAP investments."

"Together with SAP we are on a mission to make it easier than ever for you to connect SAP data to external data with confluent in real-time."

 

To get a better understanding about what Apache Kafka can to in a data and event driven world (and what maybe not), I recommend the following readings by Kai Waehner:

• When NOT to use Apache Kafka?


• The Data Streaming Landscape 2023

The interesting part will be what features will come to SAP Datasphere to support Kafka in a better way or otherwise. Currently I see nothing on the roadmap. Interesting would also be which functionality from SAP Data Intelligence Cloud goes into SAP Datasphere, as SAP DI already have a good Kafka connectivity.

 

This is just my opinion and current perspective. I’m happy to hear from you how you see these new partnerships in the context of SAP Datasphere?