Optimized Error Management and Retry Strategy with IBM MQ on SAP Cloud Integration

[ShreyasShenoy07]

Target Audience: 

This blog is primarily aimed at the SAP Integration community looking for a Retry and Reprocessing mechanism for SAP Cloud Integration, where IBM MQ is used as the message broker.

Motivation:

• As we transition to cloud-based solutions, data storage costs are on the rise. Utilizing modern iPaaS solutions with data persistence will significantly increase the operational costs for integrations. For SAP Cloud Integration (CI), there is an additional consideration of a tenant-based storage limit of 32 GB.
• We employ IBM MQ as the message broker between the third-party application and SAP CI. This message broker provides temporary persistence to handle outages and intermittent message failures, allowing us to retry messages using this persistence. The solution outlined in the blog addresses asynchronous inbound message delivery to SAP via SAP CI. 

Solution Approach:

• The core flow will transfer failed messages to the Retry Queue, appending ErrorType and Retry count headers to each message.
• A second flow (Retry flow) will be scheduled to execute at regular intervals. This flow will connect to the Retry Queue and, depending on the ErrorType and RetryCount headers, will either transfer the message to the Backout Queue or return it to the Main Queue.
• Messages designated with ErrorType= MappingError (since Mapping Errors cannot be retried directly) or with RetryCount > n (where n -> maximum number of retries) will be directed to the Backout Queue (Discarded messages). All other messages will be routed to the Main Queue(Reprocess messages) for retry attempts (ErrorType= NonMappingError, a Delivery Error that can be retried) .
• To understand this let’s consider 3 messages in the RetryQueue with the below headers the Retry flow will move them as below:
  

Core Flow

In the above flow, I did not include the mapping step. However, if you have a mapping step, the ErrorType should be set to MappingError  before the mapping step, and reinitialized to NonMappingError before delivery. This ensures the correct ErrorType is received in the exception sub-flow.

Retry Flow:

All the failed messages from the Core Flow is sent to the Retry Queue . Based on the ErrorType  and RetryCount these messages would have to be reprocessed. We will need to connect to the IBM MQ Queue manager to publish these messages to either the Main Queue (Reprocess messages) or Backout Queue (Discard messages) by picking them from the Retry Queue. For this we will need the below information (these are externalized and initialized as Headers in the flow): 

• Queue Manager Connection Details
• Queue Name (Retry Queue,  Backout Queue, Main Queue)
• Credential or Certificate for Authenticating the Queue Manager (We have used Certificate based authentication)
  

  In the retry flow Process Retry Messages is a Groovy script step which works as below:

SAP Cloud integration (CI) uses QPID JMS 0.59 for its AMQP Adapter implementation. 

1. I have created a Java Archive (JAR) file based on the same version of QPID JMS Implementation to receive and send messages from and to respective Queues (AMQP Client):
   https://github.com/apache/qpid-jms/blob/0.59.0/qpid-jms-examples/src/main/java/org/apache/qpid/jms/e...
2. The code also contains the logic to check ErrorType and publish messages to the respective Queues i.e.
   Messages designated with ErrorType= MappingError or with RetryCount > n (where n -> maximum number of retries) will be directed to the Backout Queue. All other messages will be routed to the Main Queue for retry attempts.
3. We have used TLS Authentication to authenticate our connection to the Queue Manager in the Java code. Please refer the below blog to set your certificates for authentication in the Java code:
   https://www.baeldung.com/java-custom-truststore
4. This is then called from the Groovy script Process Retry Messages. The script will essentially pass all the externalized details like the  Queue and Queue Manager connection details and the authentication details from the Security Artifacts in SAP CI to the function in the Java Archive (JAR).
   Please refer the below blog to understand how to fetch security artifacts from SAP CI:

https://community.sap.com/t5/technology-blogs-by-members/sap-cpi-fetch-all-security-artifacts-don-t-...

Pros 

• Since the failed messages are persisted outside SAP CI in IBM MQ. There is no risk  of overloading the persistence layer like in the case of alternative 2. 
• The development effort is less as the developer would only have to create the core flow and send the failed messages with the right headers. The Retry flow would only need few configurations to set it up for a specific integration. (The Retry Flow will only be built once and reused for the remaining integrations)
• Messages experiencing prolonged connectivity issues or data-related issues are transferred to the Backout queue, where they can be analyzed later.

Cons 

1. The configuration requires three queues (Main, Retry, and a Backout Queue), departing from the conventional IBM MQ setup of just a Main and Backout Queue. 
2. The script responsible for retrying failed messages may require support from respective teams in case of performance issues. 
3. Not suitable where we would need retries for very small intervals' (less than a minute). 

Request everyone to provide any improvements, suggestions  or concern wrt. to the strategy outlined in the blog.