topic Re: March Developer Challenge - CloudEvents: Week 4 in Application Development and Automation Discussions

March Developer Challenge - CloudEvents: Week 4

ajmaradiaga — Thu, 22 May 2025 06:35:17 GMT

Last week we learnt more about Event-Driven Architecture and we successfully published a CloudEvent to SAP Integration Suite, advanced event mesh (AEM). This week we will build upon what we learnt last week and we will extend our program to subscribe to a topic. By subscribing to a topic we will be able to receive messages from AEM. Before we get to the challenge, we might need to expand a bit on some concepts. Let's get started.

Week 4 - Data flow

Links to March's developer challenge:
Week 1: https://community.sap.com/t5/application-development-discussions/march-developer-challenge-cloudeven...
Week 2: https://community.sap.com/t5/application-development-discussions/march-developer-challenge-cloudeven...
Week 3: https://community.sap.com/t5/application-development-discussions/march-developer-challenge-cloudevents-week-3/td-p/13642145
Week 4: https://community.sap.com/t5/application-development-discussions/march-developer-challenge-cloudevents-week-4/td-p/13648232

Topics

Last week we mentioned that a topic is a means by which a publisher classifies a message. The topic tells us what type of message we will receive if we subscribe to a specific topic. In essence, it is a string that is composed of one or more levels. Each level is separated by a forward slash (/) and the levels can be anything. This is commonly known as topic-level granularity. The granularity allows for more targeted and efficient information exchange. Instead of having a single topic for all updates on a business object in a complex system (/BusinessPartner), the system can have distinct topics for different types of updates on a business object (/BusinessPartner/Created, /BusinessPartner/Updated, /BusinessPartner/Deleted). There is no specific schema/specification on how you need to structure your topic string but you do find that patterns are established within a system. Let's get familiar with the structure of a topic by "dissecting" a real-world topic. Below we can see a topic on which an SAP S/4HANA Cloud system will publish a Business Partner message.

Example: sap/S4HANAOD/E4L/ce/sap/s4/beh/businesspartner/v1/BusinessPartner/Created/v1:

sap/S4HANAOD/E4L: System information.
/ce: CloudEvent. We know that all events published by an SAP S/4HANA Cloud system follow the CloudEvent specification
/sap/s4: This is coming from an SAP S/4HANA system.
/beh/businesspartner/v1/BusinessPartner: Information of the Business object that we will be receiving.
/Created: This is the action that took place in the source system. In this case, it is notifying us that a BusinessPartner was created. Many actions can take place in a system, e.g. this could be /Updated, /Deleted. In another case, if we were dealing with a business object like a PurchaseOrder, there could be an event raised when it is /Cancelled or /Rejected.
/v1: Version of the message. If a new version of the message is made available, e.g. adding new fields to the payload, then this will change.

In our case, we've defined levels on our topic string based on the week, SAP Community ID and action, e.g. dev-challenge/week-4/ajmaradiaga/notification.

Now, by knowing the topic on which a message type will be published, we can create a consumer program/service that subscribes to the topic directly (aka topic endpoint) and processes the messages sent to it. Generally, you can subscribe to a topic by specifying the entire topic string when establishing the connection, e.g. sap/S4HANAOD/E4L/ce/sap/s4/beh/businesspartner/v1/BusinessPartner/Created/v1. But what if we want to subscribe to all actions (Created, Updated, Deleted) that occur on a BusinessPartner object? Luckily, in the case of SAP Integration Suite, advanced event mesh we can subscribe to the topic by using wildcards (*). For example, by subscribing to the topic sap/S4HANAOD/E4L/ce/sap/s4/beh/businesspartner/v1/BusinessPartner/*/v1 I will be able to get all messages for different actions (Created, Updated, Deleted) whose version is v1. In AEM, the > character can be used at the last level of a subscription to indicate a "one or more" wildcard match for any topics, e.g. by subscribing to the topic sap/S4HANAOD/E4L/ce/sap/s4/beh/> will bring all objects that are published under that prefix, independent of type, action, and version.

In the example above we can see how the topic level granularity can allow a consumer program/service to subscribe only to the information it needs. To learn more about wildcard characters in topic subscriptions 👉: https://help.pubsub.em.services.cloud.sap/Messaging/Wildcard-Charaters-Topic-Subs.htm

If our consumer program/service subscribes to a topic, we create a topic endpoint, and we will receive all messages for that topic subscription. That said, topic endpoints last only as long as the consumer is connected. The problem here is that our consumer needs to be online in order to receive a message. If the consumer becomes unavailable then we will end up losing messages In some scenarios, this is unacceptable and we need to ensure that we receive and process all messages published. Fortunately, there is a mechanism to retain messages without the need for a consumer service to be online 100%. Then, the consumer can process the messages asynchronously or whenever it is available. Enter Queues.

Queues

Queues allow us to subscribe to one or more topics and receive messages for all topics matching their subscriptions. The messages are received by the messaging system, saved in the queue and delivered to consuming clients if they are online and connected or held in the queue until the consumer becomes available. Queues can provide exclusive access to one consumer or access to multiple consumers where messages are distributed among the consumers. The message will be in the queue until a consumer acknowledges that a message has been processed. Only then the message will be removed from the queue.

Queue

In the case of AEM, Queues can be durable or non-durable:

Durable queues exist until they are removed by an administrative action. They accumulate messages whether clients are online or offline. When offline clients reconnect, they receive all of the messages that accumulated while they were offline.
Temporary (or non-durable) queues follow the lifecycle of the client connection and are useful for ensuring message persistence while clients are online.

Topic endpoint

As mentioned before, we can subscribe to a topic directly. A topic endpoint is created after establishing a connection to AEM (the messaging service - messaging_service.connect()) and subscribing to the topic (dev-challenge/week-4/[sapcommunityid]/processed - direct_receive_service.with_subscriptions(topics).build()). This is not a polling mechanism, but a running connection is required, through which AEM will send a message to your service. If your service is not online the message will be missed. See the code sample in the section below.

To learn more about Topic endpoints and Queues 👉: https://help.pubsub.em.services.cloud.sap/Get-Started/topic-endpoints-queues.htm

Week 4 challenge - Subscribe to topic in SAP Integration Suite, advanced event mesh

👉 Your task for this week is: Extend the program you created for last week's challenge and subscribe to the dev-challenge/week-4/[sapcommunityid]/processed topic (this will create a topic endpoint). Your program should send a message to the topic dev-challenge/week-4/[sapcommunityid]/notification and expect a response on the following topic dev-challenge/week-4/[sapcommunityid]/processed. The response will contain a hash. Please share the hash value as a comment in the discussion.

Remember that we are creating a topic endpoint, which means that before sending the notification message (dev-challenge/week-4/[sapcommunityid]/notification), you need to ensure that the subscriber on your end is working and subscribed to dev-challenge/week-4/[sapcommunityid]/processed, as the processor service will handle the notification almost instantaneously. If your subscriber is not online, the message will be missed/"lost" as no one will be listening on the topic.

Week 4 - Data flow

The diagram above explains the message processing. For example, in my case, I will publish the message to the topic dev-challenge/week-4/ajmaradiaga/notification and I will receive a response on the following topic dev-challenge/week-4/ajmaradiaga/processed. Below is an example of the response payload sent by the processor service to the dev-challenge/week-4/ajmaradiaga/processed topic:

{
  "id": "8dda9501-6379-4edc-91a7-6e78bec68746",
  "time": "2024-03-20T14:42:03.863Z",
  "type": "com.sap.dev-challenge.wk-4.processed.v1",
  "source": "https://ce-dev-challenge-wk4.cfapps.eu10.hana.ondemand.com/",
  "specversion": "1.0",
  "data": {
    "messageId": "a2e9ad2a-4955-4fb5-bd6c-91785548854b",
    "sapCommunityId": "ajmaradiaga",
    "hash": "285ed899bc5de8540a5ade05a673d60cbd68dc7bb2e21afd8507741a777e8393"
  }
}

To subscribe to the dev-challenge/week-4/[sapcommunityid]/processed topic you will need to create a topic endpoint. This can be achieved by using the Solace SDK available for your particular language. Similar to week 3, there is no need to reinvent the wheel here... check out the code available in the Solace Samples org, e.g. for Node.js, Python and the tutorials available for the different programming languages. There are plenty of examples there that show you how to use the different protocols.

In my case, I ended up using Python for which there is a library available (solace-pubsubpluis) and there is also a detailed guide for how to receive messages using the Python library. I connect to AEM using Solace Messaging. See some sample code below.

from solace.messaging.messaging_service import MessagingService, RetryStrategy from solace.messaging.receiver.inbound_message import InboundMessage from solace.messaging.receiver.message_receiver import MessageHandler from solace.messaging.config.transport_security_strategy import TLS from solace.messaging.resources.topic_subscription import TopicSubscription SOLACE_TRANSPORT_PROTOCOL = os.getenv("SOLACE_SMF_TRANSPORT_PROTOCOL") SOLACE_HOST = os.getenv("SOLACE_SMF_HOST") SOLACE_PORT = os.getenv("SOLACE_SMF_PORT") SOLACE_USERNAME = os.getenv("SOLACE_SMF_USERNAME") SOLACE_PASSWORD = os.getenv("SOLACE_SMF_PASSWORD") ... def direct_message_consume(msg_service: MessagingService, consumer_subscription: str): """This method will create an receiver instance to receive str or byte array type message""" try: topics = [TopicSubscription.of(consumer_subscription)] direct_receive_service = msg_service.create_direct_message_receiver_builder() direct_receive_service = direct_receive_service.with_subscriptions(topics).build() direct_receive_service.start() direct_receive_service.receive_async(MessageHandlerImpl()) print(f"Subscribed to: {consumer_subscription}") # Enter never ending loop to keep the receiver running while True: time.sleep(MAX_SLEEP) finally: direct_receive_service.terminate() msg_service.disconnect() ... transport_security = TLS.create().without_certificate_validation() messaging_service = MessagingService.builder().from_properties(broker_props) \ .with_reconnection_retry_strategy(RetryStrategy.parametrized_retry(20, 3)) \ .with_transport_security_strategy(transport_security) \ .build() messaging_service.connect() ########################## # Consuming from a topic # ########################## CONSUMER_SUBSCRIPTION = "dev-challenge/week-4/ajmaradiaga/processed" print("Execute Direct Consume - String") direct_message_consume(messaging_service, CONSUMER_SUBSCRIPTION)

🚨Credentials 🚨

To communicate with AEM, you will need to authenticate when posting a message. In the section below you can find the credentials required to connect.

For the adventurous out there.... I'm also sharing the connection details for different protocols that we can use to communicate with AEM, e.g. AMQP, Solace Messaging, Solace Web Messaging. In case you want to play around and get familiar with different protocols.

🔐Expand to view credentials 🔓

Connection Type:
- AMQP: amqps://mr-connection-plh11u5eu6a.messaging.solace.cloud:5671
- Solace Messaging: tcps://mr-connection-plh11u5eu6a.messaging.solace.cloud:55443
- Solace Web Messaging: wss://mr-connection-plh11u5eu6a.messaging.solace.cloud:443
Username: solace-cloud-client
Password: mcrtp5mps5q12lfqed5kfndbi2
Message VPN: eu-fr-devbroker

✅Validation process for this week's challenge ❌

As part of the validation process for this week's challenge, there is a "processor service" that will process the messages sent. The processor program follows the same validation process as in week 3 and it will generate a response, which will be sent to the dev-challenge/week-4/[sapcommunityid]/processed topic.

You can monitor the messages processed by the processor service on this website: https://ce-dev-challenge-wk4.cfapps.eu10.hana.ondemand.com/messages-hash/webapp/index.html. Similar to last week, you will also be able to see if there are any errors in the message sent, the monitoring app will tell you what the error is, e.g. not a valid CloudEvent, sapcommunityid extension context attribute missing, or sapcommunityid doesn't match community Id specified in topic. The processor service is subscribed to the topics using a wildcard - dev-challenge/week-4/*/notification and will only generate hashes for valid payloads.

Below you can see a gif of a message processed successfully by the processor service.

Re: March Developer Challenge - CloudEvents: Week 4

ajmaradiaga — Wed, 27 Mar 2024 06:48:59 GMT

Interested in learning more about SAP Integration Suite, advanced event mesh? How about you join us in this in-person event that will take place on the 6th of May @ the SAP office in Madrid, Spain - https://community.sap.com/t5/sap-codejam/event-driven-integrations-with-sap-integration-suite-advanced-event-mesh/ev-p/13648705

Re: March Developer Challenge - CloudEvents: Week 4

r00k13d3v — Mon, 25 Mar 2024 18:00:51 GMT

Here is my submission 👍

JS with solclientjs:

Results:

Re: March Developer Challenge - CloudEvents: Week 4

PriyankaChak — Mon, 25 Mar 2024 19:01:28 GMT

Hash value: db38088e6ac9ad74447faac9b0c36667df7c4edd0af2dbe4b2be939a3ad9876a

Publish:

Consume:

Re: March Developer Challenge - CloudEvents: Week 4

spirit2681 — Mon, 25 Mar 2024 18:38:27 GMT

Here is my submission:

Published message:

Received message in the consumer:

Status shown in the application:

Re: March Developer Challenge - CloudEvents: Week 4

MioYasutake — Mon, 25 Mar 2024 21:15:16 GMT

I have a problem receiving messages. The code I am using is a sample from GitHub.

https://github.com/SolaceSamples/solace-samples-nodejs/blob/master/src/basic-samples/TopicSubscriber.js

Subscribing to the topic is successful, but I cannot receive messages when I publish events to `

/dev-challenge/week-4/MioYasutake/notification`.

In the monitor app I can see my messages have been processed. I have also tested with other participants' code, but the results were the same.

Re: March Developer Challenge - CloudEvents: Week 4

r00k13d3v — Mon, 25 Mar 2024 22:58:31 GMT

Hi @MioYasutake !

I've tried with the repository you mentioned from GitHub > Codespaces, and it worked for both your topic and mine. Here is the code I modified and the command I executed:

Change Line 213:

var subscriber = new TopicSubscriber(solace, 'dev-challenge/week-4/MioYasuatke/processed');

Run command:

node src/basic-samples/TopicSubscriber.js tcps://mr-connection-plh11u5eu6a.messaging.solace.cloud:55443 solace-cloud-client@eu-fr-devbroker mcrtp5mps5q12lfqed5kfndbi2

Send request and result:

I hope this has solved your problem.

Best regards!

Re: March Developer Challenge - CloudEvents: Week 4

MioYasutake — Mon, 25 Mar 2024 23:57:00 GMT

Thank you for your prompt reply!

I discovered that I was publishing to the wrong topic - "MioYasuatke" (instead of "MioYasutake"). Correcting the topic resolved my issue.

Re: March Developer Challenge - CloudEvents: Week 4

MioYasutake — Tue, 26 Mar 2024 00:01:27 GMT

Hash value: 9fd029aeb58df50ec143d3a6f703c20e1a9f1426958cd877ebea9b6a9f99bb6f

Response:

The code I'm using is a sample from GitHub:

https://github.com/SolaceSamples/solace-samples-nodejs/blob/master/src/basic-samples/TopicSubscriber.js

How to run this code is explained by @r00k13d3v in the reply below.

Re: March Developer Challenge - CloudEvents: Week 4

ajmaradiaga — Tue, 26 Mar 2024 02:56:43 GMT

Thanks for your help figuring out the issue.

Re: March Developer Challenge - CloudEvents: Week 4

xavisanse — Tue, 26 Mar 2024 11:51:54 GMT

Yep guys. I've tried it and doesn't works 😫 done in node.js. ~~I submited also the .env file to git if anyone wants to check it~~. I dunno. Appears at subscribed and listening, then i trigger the message to be consumed but nothing happens.
git repo - https://github.com/xsansebastian/codejams
environment entries here 🔐 ⬇️

#SOLACE VARIABLES ENV WEEK4

SOLACE_HOST=tcps://mr-connection-plh11u5eu6a.messaging.solace.cloud:55443

SOLACE_USERNAME=solace-cloud-client

SOLACE_PASSWORD=mcrtp5mps5q12lfqed5kfndbi2

SOLACE_VPN=eu-fr-devbroker

#SOLACE VARIABLES WEEK3

SOLACE_HOST=mr-connection-plh11u5eu6a.messaging.solace.cloud

SOLACE_PORT=9443

SOLACE_USER=solace-cloud-client

SOLACE_PWD=mcrtp5mps5q12lfqed5kfndbi2

I first executes week4, then week3. the message of subscription is now active corresponds to

sessionEvent SUBSCRIPTION_OK

Thanks in advance for your help 🙂

Re: March Developer Challenge - CloudEvents: Week 4

geek61 — Tue, 26 Mar 2024 12:02:16 GMT

Borrowed heavily from:

https://github.com/SolaceSamples/solace-samples-nodejs/blob/master/src/basic-samples/TopicSubscriber...

Processed through Event service:

Processed response:

[11:54:49] Received message: "▼☺{ "id": "1e1e10dd-0302-4f9b-b785-6fa84a89e43e", "time": "2024-03-26T11:54:49.603Z", "type": "com.sap.dev-challenge.wk-4.processed.v1", "source": "https://ce-dev-challenge-wk4.cfapps.eu10.hana.ondemand.com/", "specversion": "1.0", "data": { "messageId": "6caee628-04a1-4565-ae1e-8eb6d4370c96", "sapCommunityId": "geek", "hash": "e92ac42a4f66aaf0665d9254d2494efe0f6a1c9d73e25029ec1c2b680d06b465" } }"

Re: March Developer Challenge - CloudEvents: Week 4

r00k13d3v — Tue, 26 Mar 2024 12:04:10 GMT

Hi @xavisanse !

I think you are calling the topic for week 3, you should call the one for week 4 (dev-challenge/week-4/${SAP_COMMUNITY_ID}/notification).

I'll leave you the link to the monitor so you can see it when you make the request.

See if that works for you.

Greetings!

Re: March Developer Challenge - CloudEvents: Week 4

xavisanse — Tue, 26 Mar 2024 12:08:56 GMT

absolutely true 🙂 it happens when you are lazy coding in easter holidays 🙂 and reads in diagonal. now works 🙂 kudos for you

Re: March Developer Challenge - CloudEvents: Week 4

xavisanse — Tue, 26 Mar 2024 12:13:31 GMT

awesomeCode:

hashValue = 8a48e62638474227a18d5e4cf0640b22486a513f0fd0e17c95f9d191bcea3ad1

Console Output:

Re: March Developer Challenge - CloudEvents: Week 4

Ruthiel — Tue, 26 Mar 2024 14:56:40 GMT

Hello @ajmaradiaga !
Thanks a lot for this great opportunity to handle those CE!

Here is my submission for this last week:

The hash code is:

90128999a98108464d992819bbfd01941afa25ec90c64089663498411af037fe

Re: March Developer Challenge - CloudEvents: Week 4

emiliocampo — Tue, 26 Mar 2024 19:08:17 GMT

"hash": "26105dec098dd4ee5410f3ba27e7f755f23551851888fc76f36364557d461b58"

Publish:

Consume:

Hola @ajmaradiaga, thank you for this challenge! I'm not a pro-developer and it's been difficult for me, but by going through all the documentation, I've been able to do it 😃!!! Gracias por informar del evento en Madrid!

Re: March Developer Challenge - CloudEvents: Week 4

ajmaradiaga — Wed, 27 Mar 2024 06:44:14 GMT

Great! As mentioned in the post... there is no need to reinvent the wheel.

Re: March Developer Challenge - CloudEvents: Week 4

ajmaradiaga — Wed, 27 Mar 2024 06:47:28 GMT

@emiliocampo, gracias por participar en el challenge! En el evento vamos a familiarizarnos con AEM y tambien habra algo de codigo :-). Nos vemos el 6 de Mayo?

Re: March Developer Challenge - CloudEvents: Week 4

ajmaradiaga — Mon, 01 Apr 2024 03:24:38 GMT

All valid submissions for week 4 so far...