BTP Event Mesh Advance Concepts

[showkath_naseem]

SAP BTP Event Mesh (aka enterprise-messaging) is a fully managed cloud-based service that helps connect applications, services, and systems across hybrid environments. It allows the exchange of asynchronous events between components and distributes business events from both SAP and non-SAP sources across hybrid landscapes in real-time, using event-driven communication. The key benefit of this approach is that the components can operate independently and do not have to wait for a response before proceeding, which can significantly increase the overall performance and scalability of the system.

The main purpose of the Event Mesh service is to enable the publication and consumption of business events between back-end systems and extension applications that are built on SAP BTP. The service is based on a publish-and-subscribe model, where events are published by a sender and can be received by interested subscribers in a decoupled manner. This design allows components to communicate with each other without being aware of one another, which enhances flexibility and reduces interdependence between components.

 

Key Benefits :

Some of the key benefits of using BTP Event Mesh include:

1. Scalability: SAP BTP Event Mesh can handle a large volume of events with low latency and high throughput, making it suitable for high-performance and mission-critical scenarios.SAP BTP Event Mesh allows for the scalability of event-driven applications, allowing for the addition of new services and components(consumers) without the need for manual intervention.
   you can easily add more capacity when needed.


2. Flexibility: SAP BTP Event Mesh allows for the flexibility of event-driven applications, allowing for the addition of new services and components without the need for manual intervention.
   Additionally  It supports various event formats, protocols, and endpoints, including HTTP, AMQP, MQTT, JMS, SAP Message Server, SAP Event Mesh Service, and SAP Event Bus. This makes it easier to connect applications and services with different technologies and protocols.


3. Reliability: SAP BTP Event Mesh provides a resilient messaging infrastructure and secure event-driven architecture, allowing for the reliable delivery of events across multiple services and components, that is designed to ensure high availability.


4. Decoupling: It enables loosely coupled integration between applications and services, which reduces dependencies and increases agility. This means that changes made in one application or service do not affect the functioning of other applications or services.


5. Improved performance: Event Mesh helps to reduce latency and improve throughput by providing a distributed messaging infrastructure that is optimized for high-performance event-driven applications.


6. Security: SAP BTP Event Mesh provides an additional layer of security for event-driven applications, ensuring that only authorized users and services can access and process events.It provides secure and reliable communication through end-to-end encryption, access control, and message validation.


7. Monitoring and Management: It offers comprehensive monitoring and management capabilities through the SAP Cloud Platform cockpit, including event tracking, tracing, and error handling. This allows developers to troubleshoot and resolve issues quickly and easily.


8. Cost Savings: SAP BTP Event Mesh helps to reduce the cost of development and maintenance of event-driven applications,by eliminating the need to maintain and manage separate messaging infrastructure for each cloud provider.

Overall, SAP BTP Event Mesh provides a flexible, scalable, and secure solution for event-driven communication that can help organizations streamline their application integration and improve their business agility.

Synopsis :

So far you might know that in an event-driven architecture consists of event producers that generate a stream of events, and event consumers that listen for the events.

Publisher / Producer	A application/system/component  which will add message/request in the queue when when configured event occur
Consumer/Receiver	A application/system/component  which will receive/process the message from the queue
Queue	A queue is a storage  space used to host a message until it’s received by the subscriber. Queue wont process message, it is first consumer & just store message/event.   You need to explicitly create a queue using Admin UI or Management REST API.
Topic	A topic is a space used to host a message for a specified period of time defined by the sender. Queue is managed object but Topics are not managed. You can say that a topic is created on the fly when you start publishing messages to a topic, and destroyed when there are no consumers listening to that topic. If a publisher sends an event to a topic and there are no subscribers for this topic, then there is no storage for any period of time.
Queue Subscription	A queue subscription allows a queue to subscribe to a topic for its message retention capability.
Message Client	A Message client is the central component in BTP EMS to define the access from / to Event Mesh, It also controls the access to topics and queues

1)  For Error handling Scenarios what are options provided in SAP Event Mesh ?

 

From Producer Side you have below options

Request Header	Values	Description
x-qos	0 1	Quality of Service (QoS) Quality of Service denotes the way the messaging protocol handles retransmission of the message and ensures its delivery regardless of network reliability.   The allowed values for specifying QoS are 0 and 1. The API for publishing messages requires a mandatory header x-qos. When the client calls the API for publishing messages with QoS 0, the service tries to deliver the message and returns an HTTP response with code 204 irrespective of whether the message is delivered. The best effort is made to deliver the message but, doesn’t guarantee that the message is sent to queue or topic. When QoS 0 is used, the message is deleted from the queue without an acknowledgment from the client.   If a message client calls the API for publishing messages with QoS 1 for guaranteed delivery, then, the service responds with the HTTP response code 204. If the 204 response code isn’t received, it’s the client’s responsibility to retry until the response code 204 is received.  The message is deleted from the queue only after the acknowledgment is consumed   Use QoS=1, as you want the message to be processed and "acknowledged" by EM As of now EMS providing a QoS=1, which means at least once. How to do that is dependent on the used protocol. Exactly once in order is not supported out of the box. This requires additional efforts on sender and on consumer side. The QoS is a contract between sender and broker and is a second contract between broker and consumer, that's one reason why this is not supported out of the box. More details on QOs https://help.sap.com/viewer/bf82e6b26456494cbdd197057c09979f/Cloud/en-US/3f424ff1ae3b4bc084c4f1ea0be...
Time to Live		Attribute in Queue With every publish request, the message client can set a time to live for the message in milliseconds. If the API is called without this header, the default value of 2592000000 (30 days) is used. The Time-To-Live (TTL) value (in milliseconds) can be set for each guaranteed message published by a message producer. Endpoints can be configured to respect message TTLs through the REST APIs for management or the user interface. If message TTLs aren’t honored, the message is discarded by the endpoint

 

From BTP Event Mesh you have below options while configuring queue

 

BTP Event Mesh offers below options in case of Error scenarios

Example :  Lets assume Target System is down in your ECO system  or when any errors occur in consumers (For example CI Iflow ) & you configured DeadMessageQueue(DMQ) to store messages

DeadMsgQueue(DMQ) 

DMQ is also just a queue.

> You can have messages go to a DMQ once the consumer has rejected them x-times...

> or you can move them to a DMQ because it hasn't been processed within x seconds after the event has initially arrived

Then you need to decide depending on your scenario, how to handle the DMQ afterwards

So far , SAP Event Mesh doesn't offer any generic way to process a DMQ

You can have client to Processing the DMQ or  it might be the same client. But someone needs to connect to that queue and handle the events.

One reason for this is that the consumer needs to judge why a message has ended up in a DMQ and if it makes sense to re-process it. If the original consumer was offline, it makes sense to re-process a message, but what if the original consumer actively rejected a message because of e.g. an invalid payload it doesn't make sense to send it again. How to handle the order of messages in a re-processing scenario? All this and more needs to be decided individually for each use-case.

 

Parameter	Values	Description
deadMsgQueue		deadMsgQueue  is the name of the Dead Message Queue (DMQ) used by the Queue. By default, each Queue is assigned a Default DMQ which does not exists. User must create a DMQ and assign it to the Actual Queue. when TTL expires for a message or the Number of redelivery attempts for a message exceeds the Max Redelivery value the messages instead of getting discarded can be moved to DMQ EMS supports dead letter queues. The configuration needs to be done through the management API: https://help.sap.com/doc/75c9efd00fc14183abc4c613490c53f4/Cloud/en-US/rest-management-messaging.html... * you can also use the management API . https://help.sap.com/doc/75c9efd00fc14183abc4c613490c53f4/Cloud/en-US/rest-management-messaging.html...
Max Redelivery Count	Default : 0 Minimum value : 0 Maximum value : 255	This parameter you can configure in Event Mesh Message Client à While creating/Updating Queue   1) For example, If you set Max Redelivery Count to 3, that means EM will "push" the message maximum 3 times to the consumer if the consumer actively rejects the message. Technically the consumer has 3 options: "Accept", "Release" & "Reject". This redelivery count is only increased on a "Reject" If consumer “Release" in case of CI(aka HCI) -internal error, you'll get that message inifinitly. 2) Either when the Max Redelivery Count is reached OR when the Max TTL is expired, the message will be removed from the queue. If you'd specify a Dead Message Queue (DMQ), it would be moved there.
Max Time to Live [in Seconds]	Default : 0	When Respect Time to Live              = ENABLED while creating/updating Queue in Admin UI Cockpit Maximum time in seconds a message can stay in the Queue when Respect TTL option enabled. A message expires when the lesser of the sender assigned time-to-live (TTL) in the message and the maxTtl configured for the Queue, is exceeded. The expired mssage will move to Dead Message Queue (DMQ). A value of 0 disables expiry. More information on help portal.

 

From Consumer Application

When QoS = 1 & If the 204 response code isn’t received, it’s the client’s responsibility to retry until the response code 204 is received.  The message is deleted from the queue only after the acknowledgment is consumed

 

If Receiver application is CPI Iflow then while configuring the AMQP Receiver Adapter you have below options in retry mechanism

 

Max. Number of Retries	The default is set to 0, meaning there are no retries executed and the delivery status you defined for Delivery Status After Max. Retries is sent directly to the message broker. The maximum number of retries can't exceed 99. If you do not specify the number of retries, endless retries are executed. In this case, the delivery status you defined for Delivery Status After Max. Retries will never be sent.	Define the number of retries to be executed before a different delivery status is sent to the message broker.   https://help.sap.com/viewer/368c481cd6954bdfa5d0435479fd4eaf/Cloud/en-US/99ce6748400b4dbfbadf633aeb1...   Blog Post https://blogs.sap.com/2019/11/04/sap-cpi-retry-send-failed-asynchronous-messages-based-on-time-inter... See this blog for a summary of the capabilities supported by the different brokers. https://blogs.sap.com/2019/11/20/cloud-integration-connecting-to-external-messaging-systems-using-th...
Consume Expired Messages	By default, this option is deactivated.	Select if the adapter is to consume already expired messages.
Delivery Status After Max. Retries	Define one delivery status that is to be sent to the message broker after the maximum number of retries. Choose between the following options: ·         REJECTED ·         MODIFIED_FAILED_UNDELIVERABLE

 

 

2) Size & Limitations

 

Please check the limits on help portal and SAP Event Mesh (default plan) Size and Limits. Note that size restriction applies to request payload as well as headers.

For more descriptions of the messaging resources please check https://blogs.sap.com/2021/02/22/event-mesh-resource-units/

 

 

• Question : Why The maximum message size is 1 MB for all messaging protocol ?

Generally, EMS Designed to send maximum message size of 1 MB for all messaging protocols because EMS not designed to transport huge data between components.

If you need additional data from Provider then Consumer applications ( Example iflow ) has to initiate a synchronous call to fetch additional data from Provider

 

3) How EMS Support Performance improvements

In EMS Event-Driven Architecture (EDA)  Sender doesn’t need to ask the consumer all the time if there are processing resources available, or if some task is completed by consumer. Sender works in its own speed & consumer works in own speed, none of these  EDA components black each other.

For better performance in your EDA consumers should be quicker & available i.e  If consumer is always online then should not be message in queue .

Queue just store message so it is responsibility of Consumer in EDA to process message

 

4) Difference between Topic & Queue

#	Queue	Topic
1	Queue is a managed object, i.e. you need to explicitly create a queue using UI or Management REST API.	Topics are not managed. You can say that a topic is created on the fly when you start publishing messages to a topic, and destroyed when there are no consumers listening to that topic.
2	Queues are persistent. When there are no consumers to a queue, messages are stored in the queue.	Topics are not persistent. If your consumer to a topic is not running when a message is published to the topic, you'll never receive that event.

 

5) Can two applications/consumers listen to Same Event Mesh Queue

Only one consumer receives any message from a queue.

By default, queues are created in NON_EXCLUSIVE mode, i.e. if there are more than one consumers to the queue, messages are delivered to each consumer in a round-robin fashion.

You may also create an EXCLUSIVE queue, in that case the first consumer on the queue receives all messages. If the first consumer disconnects, the second consumer starts receiving all messages, and so on.

 

You can have multiple queues/consumers subscribe to the same topic.

That is one of the key features of an event driven architecture: the sender doesn't know it consumers!

 

More information on help portal.

 

6) Cardinality

 

In general each consumer has its own queue , but all can subscribe to one topic. (Please read above point 5)

 

• 1 Event Provider can publish to one Topic                                                       .    =        1:1


• One Topic can be attached to  n-number of Queues                                            =         1: M

i,e you can have multiple queues/consumers subscribe to the same topic

• Multiple Queues can be subscribe to one topic                                                      =         M:1

 

i.e you can create separate queues (one for each of your project or iflow ) that are all queues subscribed to topic on which your business object events are published i.e each of queue gets a own copy of messages published to that particular topic & there wont be message lost , no competition for the message between projects or iflows /consumer .

 

• One consumer = One different queue [different name in each CI Iflow AMQP)     =         1: 1

 

• However one consumer app can also subscribes to multiple queue (BTP APP can can have custom source code to consume/listen from multiple Queues ).At the end ,it is up to producer or consumer requirement


• Instead of creating one new queue for each CI iflow the other possible approach is , you can also have generic iflow subscribe to single queue, determine what kind of event from payload & trigger other subprocess/sub iflow. Having single generic iflow & one Queue In this case you will reduce event mesh connection but may be maintenance/tracking will be difficult

Note :  Here you can create one unique message client for each type of consumer & distribute event mesh credentials securely to consumers of your eco system.

7) Is Producer has to publish to Topic or Queue ?

 

If your ED architecture demands PUB-Sub model i.e one Produce & many consumers then Publish to Topic

Topic is for publish-subscribe scenarios, i.e. one-to-many messaging. Every subscriber to a topic receives all messages published to that topic.

Where as Queue is for point-to-point messaging. A message can be consumed by one and only one consumer.

 

😎  Should consumer consume from Topic or Queue

Recommendation is : Consume from Queue

Explanation :

It shouldn't matter if consumer consume from the topic or from the queue. The queue can be considered like the "first consumer" that is attracting the configured topics. So it's just one additional consumer. Each consumer get it's own copy of an incoming event. No matter if it is 0,1 or 1000 consumers for the same event. This also should happen at the same time. You should not see any delay on the EM side (there could be delays due to the implementation of the consuming app).

In general Event Mesh recommend that consumers for example  BTP app should also consume via a queue. If your BTP app is down for maintenance reasons or network issues, it will loose events that are sent during this downtime. Maybe okay for a demo scenario, but not for a productive environment.

 

Example

Weather APP , Temperature app connected to Event Mesh online

if you lose few message then It does not matter  & I,e it won’t effect business

In this case you listen from topic . I,e When your consumer app offline does not worry or not have requirement to read lost messages when it was not online .

 

Banking APP : Where every transaction is important .Consumer APP don’t want to lose messages .So You read message from Queue from consumer APP ,because message will be stored in Queue even when consumer is offline .

Once Consumer APP is active again AMQP Adapter reads buffered message form Queue automatically. No need to trigger again neither from Provider nor consumer has to start .As consumer established connection with Event Mesh it will read messages form Queue when it comes online .

 

Sometimes CI tenant could be down or during Iflow deployment or restart ,some one stopped , in all these outage/consumer downtime cases , consumer still have an option as soon as it came online iflow/consumer app (AMQP supported)  can read message from Queue .

9) What is the recommended way to connect Cloud Integration  (aka CPI)  with Event Mesh

Recommended way to connect to CI iflow to receive messages from Event Mesh is to user AMQP Adapter

Please checkout  bog post describing the details: https://blogs.sap.com/2019/11/20/cloud-integration-connecting-to-external-messaging-systems-using-th...

10) What are best practises for Messaging Paradigms

Consider to always apply the following messaging paradigms to your messaging / eventing scenario.

• Every Event in Publisher to be subscribe to One Topic,

For example

Event-A Publish to Topic-1,

Event-B Publish to Topic-2

i.e different Events to be Different Topic

• Every Consumer in EDA listen to one Queue i.e  Create N queues for the N receivers.

For example

Queue-1 is for Consumer-1

Queue-2 is for Consumer-2

Create N queue subscriptions for the queues created above and assign them to the same topic name.

• Publish messages to topics only: as soon as the messaging traffic leaves the tenant boundaries (e.g. Tenant/Reuse Traffic) each EM participant shall publish messages to well-defined topics only. P2P messages (send messages to queues directly) shall be avoided in any case. Reasons are the following:

• 
  
  
  
  • 
    
    
    
    • Multicast of messages → it is up to the consumer to subscribe 1...n to a certain topic
    
    
    • Scaling flexibility → queues lead to a stickiness of consumers to the corresponding broker resource which would prevent EM to optimize the virtual broker
    
    
    • Clear Event/Message contract → topics are the basis to describe the event contract and events can be identified clearly through their topics in the future event catalog (Concept Eventing).
    
    
    • Non-existent consumers → the approach avoids to deliver messages to queues w/o attached consumers.
    
    
    
    
  
  
  
  

• Publish to your own namespace: never publish to foreign namespaces


• Consumption-driven subscriptions:the consumer defines how and how often a message is consumed

11) Example , Naming Conventions, Rules

In Publisher/Sender application, let’s assume you have below Events

• OrderCreated


• PaymentRecived


• OrderConfirmed


• Shipped


• Refunded


• OrderFailed

 

Dependents on your business & architecture ,the N number of consumers (Example : Application1,Application2 , CI-CPI-Iflow1,CI-CPI-Iflow1,) would like to receive notifications

So you can create N number of Queues i.e Every Consumer in EDA listen to one Queue. For example "Queue-1" for Consumer-1 similarly for "Queue-2 for consumer-2"

If one consumer application-1 intersted in 3 events , for example the can create 3 queues

 

<namespace>/OrderCreated

<namespace>/OrderConfirmed

<namespace>/OrderFailed

 

 

Queue Naming Conventions & rules :

 

The total length is limited to 100 characters.

All queue names must have a namespace as a prefix

The queue name must consist of one or more characters such as "A to Z", "a to z" or "0–9 ", “.”, “-”, “_”, and “/”.

 

Topics : Example

You can create Topics as below

<namespace>/OrderCreated <namespace>/OrderConfirmed <namespace>/OrderFailed

or My recommendation would be: <namespace>/Sales/Order/Created <namespace>/Sales/Order/Confirmed <namespace>/Sales/Order/Failed   Advantage is With this you could have queues subscribing to: <namespace>/Sales/* ---> would get all "Sales" related events <namespace>/Sales/Order* ---> would get all "Sales Order" related events <namespace>/Sales/Order/Failed --> if your queue is only interested in "Failed Orders"

 

The syntax for topic names is as follows:

Topics must consist of one or more segments.

Forward slash "/" (ASCII 0x2F) must be used as a segment separator.

The segment separator must not appear at the beginning or ending of a valid topic.

The segments must consist of one or more characters such as "A to Z", "a to z" or "0–9 ".

Up to 20 segments can be used.

Empty segments aren't allowed.

The total topic length is limited to 150 characters.

 

 

The following are examples of valid topic names:

 

<namespace>/Production/Confirmation/Created

 

<namespace>/Process/Order/Released

 

<namespace>/Sales/Order/Created

 

 

The following are examples of invalid topic names:

//Production/Order/Released

<namespace>//Order/Released

<namespace>/Sales/Sales Order/Created

 

 

For Constraints Refer : https://help.sap.com/viewer/bf82e6b26456494cbdd197057c09979f/Cloud/en-US/72ac1fad2dd34c4886d672e66b2...

 

12 ) Is BTP Event Mesh Supports “Basic “ Authentication ?

 

Answer is ) No , EMS Supports only oAuth

13) What are options available in EMS to Delete Messages in Queue

A queue is a storage space used to host a message until it’s received by the subscriber.

Queue wont process message, it is first consumer & just store message/event.

So how to clean messages from queue  , So far as I know we have below options

 

1) EMS Supports Timeout Handling for Message in the Queue

This option you can configure in Admin EMS  Web UI , while creating or editing queue

When Respect Time to Live              = ENABLED while creating/updating Queue in Admin UI Cockpit

Maximum time in seconds a message can stay in the Queue when Respect TTL option enabled. A message expires when the lesser of the sender assigned time-to-live (TTL) in the message and the maxTtl configured for the Queue, is exceeded. The expired mssage will move to Dead Message Queue (DMQ). A value of 0 disables expiry.

More information on help portal.

 

2) Purge Queue

In EMS Admin WebUI , Queue has "Purge Queue" option to delete all messages in Queue

14) Monitor Resources

 

EMS Admin WebUI has various option to configure Quota, Size of Resources.                                    UI also provides option to Monitor Consumption as shown below

15) What are Development ,Implementation Options in EMS

 

EMS Applications you can develop using protocol agnostic libraries that can be used at the application configuration level for Java and Node.js

 

Build Applications Using Java

 

To build messaging applications using Java, see Samples on GitHub for Java Information published on non-SAP site.

http://help.sap.com/disclaimer?site=https://github.com/SAP-samples/enterprise-messaging-client-java-...

Build Applications Using Node.js

To build messaging applications using Node.js, see Samples on GitHub for Node.jsInformation published on non-SAP site.

http://help.sap.com/disclaimer?site=https://github.com/SAP/enterprise-messaging-client-nodejs-sample...

Use REST APIs

To use the REST APIs provided with the service, see REST APIs for Development.

EMS Provides REST for Development , Testing .

Use REST APIs to Send and Receive Messages

Use REST APIs to Manage Queues and Queue Subscriptions

REST APIs for Messaging

https://help.sap.com/viewer/bf82e6b26456494cbdd197057c09979f/Cloud/en-US/dc18d8ebc226404c87db22e818f...

 

Thank you for reading! If you enjoyed this post, please consider giving it a ‘'Like' ,'Share'  it with your friends and followers, and 'Follow' me for more content like this. Your support means a lot to me!

About me :

LinkedIn > Showkath

Happy Eventing 🙂 !