Reversing the time with Advanced Event Mesh: Unleash the Power of Message Replay

[BarisBuyuktanir]

As mentioned in various blog posts and other articles, SAP BTP currently have 2 different solutions in the area of Event Driven Integration since Advanced Event Mesh joined the family last year with  much more offerings and capabilities.

I will not be discussing each of them as christopher.aron  in his blog make a very good comparison between these two (Event Mesh and Advanced Event Mesh).

Contrary to their similar names, Advanced Event Mesh and Event Mesh are actually two distinct solutions (both have different reasons to exist) . When it comes to perfomance and usability, Advanced Event Mesh obviously outshines Event Mesh offering a greater number of enterprise scale features.

This is why many customers start to choose and take advantage of various capabilities offered by the Advanced Event Mesh. In this blog post, I will discuss a significant feature that can provide great benefits once designed to be used effectively: "Message Replay."

What is Message Replay

Message Replay in SAP Advanced Event Mesh is a feature that allows the event broker to resend messages to clients, upon request. This can occur significantly after the event broker's initial reception of the messages, like a journey back in time.

When enabled, persistent messages are stored in a “replay log” and they are retained until the log reaches its capacity within the event broker.

In message replay, you have the option to request

• all messages in the replay log,


• messages following a specific “replication group message ID”,


• messages starting from a specified replay start time.

FIGURE 1 – Message Replay Flow

The idea is simply: When you lose your messages as a consumer -which means they are not available in your queue for some reason (not that Advanced Event Mesh lose it, but you as a consumer acknowledged that you receive and consumed it and then you lose it from the consumer end). Even in this case Advanced Event Mesh could bring you the messages again to your queue if it’s configured to do so.

What happens when Replay is initiated?

When you initiate Replay on a queue

• Existing messages are removed from the queue and older messages are replayed to that queue.


• Once all the older messages have been replayed and consumed, the consumer automatically starts consuming the live stream of messages.

 

What are some of the use-cases & what can you achieve with Message Replay

Imagine a scenario where an application is consuming the messages from Advanced Event Mesh and acknowledging the consumption; therefore messages are deleted from its queue as usual. After a while application's records are also corrupted for some reason and it requires the messages again. If the replay log is enabled, the consumer’s queue could be filled up with the exact records and the application can consume the same messages again without affecting other consumers/producers.

Another scenario would be the late joiners where there are applications already consuming the messages from event broker however a new application would like to consume the messages published before it subscribes to a topic or connects to Advanced Event Mesh late. (Situation like "get me the messages since Jan 2023 until I join the club on March 2023"). In this case these messages could be replayed to its queue and then it can continue consuming the regular ones as usual.

These are only two very generic cases among others where we can utilize message replay. Keep in mind that we need to use this feature carefully as it consumes resources in the event broker. Technically it’s another queue in the event broker with a size limitation therefore while designing the system as a whole you should be “selective” when it comes to which messages to be chosen for replay.

Enough for the information.. Let’s use the feature with an example.

First things is to enable Message Replay. For this just Open the Broker Manager, on the left select Replay. Determine the size for your Replay log, keep the topic filtering for further filtering requirements and that’s it. In this scenario I choose 400MBs, but in real scenarios you can pick a much larger size.

FIGURE 2– Message Replay (parameters)

Business Scenario

Imagine a scenario where Advanced Event Mesh receives orders from S/4HANA system and publishes them to multiple different applications. In our case there are two different applications (consumers) awaiting for the order information.

• Regardless of the country, Application 1 is interested in all orders.


• Application 2 is a specific one, interested in the orders from United Kingdom and Turkey.

These two applications receive and consume the orders from their queue once the orders are published and once acknowledged by them, as usual the orders are deleted from their queue.

Normally we might be enabling to activate the replay for “all” orders published, but for the sake of demonstrating topic filters usage, we will be configuring the replay for only “important” orders. (In real life resources are not limitless, most probably we won’t be storing everything even if we could as well)

The very basic topic structure for the demo is as follows:

TOPIC : orders/{countryCode}/{isImportant}

• So the orders are published to topics such as
  
  
  
  • 
    

    orders/uk/1            (important order uk)

    
    
  
  
  • 
    

    orders/uk/0            (not important order uk)

    
    
  
  
  • 
    

    orders/us/0, orders/tr/0 etc.

    
    
  
  
  
  

In our scenario Application 2 has received the orders it’s interested and acknowledged them as usual, but because of some technical problems it has lost the messages from its database and it needs to reprocess them again.

Thanks to the Message Replay capability of Advanced Event Mesh, we can provide them the messages again, since the beginning / for a particular timeframe / or from a particular message.

However by our design, not all the messages / all the orders will be re-delivered. Only the ones that the application is subscribed to (A)+ within what we promised to be stored for the “replay” (B)

For (B) we are using topic subscriptions for replay. We only promise to store important orders for replay therefore our replay subscription would be as follows:  (meaning orders regardless of country but marked as important)

FIGURE 3– Subscriptions for Replay

In this way only the important orders are stored in the replay queue.

FIGURE 4– Queues for Application 1 and 2

 

Now I created 2 queues as usual to store my persistent messages until the applications consume them.

FIGURE 5– Subscriptions for Application 1

FIGURE 6– Subscriptions for Application 2

Application 1 is interested in all the orders, while Application 2 requires orders from Turkey(tr) and United Kingdom (uk) both regardless of its being an important order or not.

First of all let's publish 6 orders like the ones below.

• important, country : TR


• important, country : UK


• important, country : US


• not important, country : TR


• not important, country : UK


• not important, country : US

In this case I would expect my first queue to have 6 orders, and my second queue to have 4.

FIGURE 7– Number of messages in the queues after 6 messages (orders)

What about my replay queue?

The answer would be 3. which means even if they are lost for the consumer side; remember I decided to be replaying the important orders.

FIGURE 8– Number of messages in the replay log queue after 6 messages (3 important orders)

Now it comes to the tricky part. Application 2 consumes the messages from my its queue and acknowledged that it consumed and stored the data. So its queue becomes empty but then it notices that its database is corrupted, therefore it loses all 4 of its orders, including the 2 important ones.

FIGURE 9– Number of messages in Application 2 queue after it consumes all the messages 

 Now at least they request to get the important orders from Advanced Event Mesh again. We don’t need the publisher, S/4HANA to resend the order, but we replay their messages to Application 2’s queue. The magic is not all, but the important orders will be pushed; and only the particular ones they are interested in; in our case the important orders for Turkey and the UK.

We initiate “message replay” for the Application 2’s queue by clicking on the queue, in the summary tab, from the actions, choose start replay.

FIGURE 10– Start Replay for the queue

As mentioned we can prefer getting every message since the beginning(from the oldest one that we have), since a particular date-time or starting from a particular message until the end. In this case I didn’t enter a time or ID therefore I prefer all important orders for Application 2.

FIGURE 11 – Options for Message Replay

This message is important as it indicates that any message in the queue will also be deleted. (as a trick you can use another queue for the replay messages)

Once confirmed, Voilà!,  you have your “important” orders in your queue again, as below. You didn’t lose your important orders

 FIGURE 12– Application 2’s queue after message replay

Just remember that Message Replay Log is simply queue within Advanced Event Mesh, although it's a very useful system-managed one. In order to utilize the resources, there is also an option for the “trimming of messages” which is basicly deleting of the messages from the replay log to increase the usable capacity manually when log grows larger and fills up. This option could be used by systems operators when the size grows to certain threshhold values.

There are also some tricks that you can do while you’re “saving”(replaying) your lost messages, thanks to the powerful topic logic of Advanced Event Mesh with the subscriptions. A good design could help you a lot: One example would be putting some smart IDs in the topic string and subscribing to only for those while replaying the messages, but there might be many examples depending on your scenario, which saves you from a lot of trouble when required.

This was only a quick/simple demonstration of “Message Replay” capability of Advanced Event Mesh.

Although the scenario mentioned is very basic(as well as configuring it in AEM), you can extend it to more different use-cases to recover from hard situations without affecting the publisher and other consumer of the events.

 

More to come for demonstrating the features of Advanced Event Mesh in my future posts..