How to pass Properties from SAP AEM to Kafka Headers & build dynamic AEM Topics from Kafka Headers

[AlexPfeil]

Introduction

In this Blog by Florian the Connectivity between SAP AEM and Kafka is explained. There, in the first step, we have 2 connections

AEM Queue SAPBlogKafkaSenderQueue -> Kafka Topic SAPBlogKafkaSenderQueue

Kafka Topic SAPBlogKafkaSenderQueue -> AEM Topic WelcomeSAPEvents

In this blog we want to have a look at Dynamic Topics in AEM. SAP AEM offers the possibility to have dynamic filters, so Consumers can already filter in their subscription to the topic based on certain values. 

 

Kafka Sender

The Queue SAPBlogKafkaSenderQueue from Florians Blog does not have a Topic Subscription. In my AIFAEM Blog I have created a scenario to connect a S/4HANA System to AEM with AIFAEM. I´ll use the topic from there:

The Topic of my Test Business Partner is "sap/businessPartner/0001/0000000181". "0001" is the Grouping of the Business Partner, "0000000181" is the BusinessPartner Number this could be different for other Business Partners.

In my Custom Header Blog I have explained how to pass Custom HTTP Headers to AEM. Now we want to pass this information to Kafka. Since Kafka does not support dynamic Topics, we need to store the Headers in the Kafka Headers. User Properties are passed to Kafka automatically and do not need to be specifically customized.

 

Kafka Receiver

Now when receiving the Event back from Kafka, we want to pass the Header Values to the AEM Topic dynamically. 

Currently the Topic Binding for the Kafka Receiver only passes the Events to a static AEM Topic "SAPAEMInternalEvent"

Let´s change it to 

"KafkaBusinessPartnerReceiver/${kafkaHeaderAsString("businesspartnergroup")}/${kafkaHeaderAsString("businesspartnernumber")}"

 

Test

Now If I change a BusinessPartner in my S/4HANA System, the Event should be published to AEM Topic "sap/businessPartner/0001/0000000181", persisted by our Queue "SAPBlogKafkaSenderQueue" which is consumed by our Kafka Sender posting it to Kafka Topic "SAPBlogAEMKafkaSender". This Topic we consume from AEM with out Kafka Receiver and post it to AEM Topic KafkaBusinessPartnerReceiver/0001/0000000181.

For Testing Purposes I have created a BTP Integration Suite Iflow consuming KafkaBusinessPartnerReceiver/>

Here we can see what Kafka Topic it was (Header "UserProperties", Property "k_topic") and which Topic was created in AEM (Header Destination).

Conclusion

This blog demonstrated how AEM Message Properties are passed to Kafka and how we can build up dynamic AEM Topics based on Kafka Headers. This, for instance, can make filtering of Kafka Events possible without having to include BTP Integration Suite in between. There are more possibilities to explore for the Substition Expressions, documented here.