Implementation of Exactly Once in Order (EOIO) scenario in Integration Suite

[pdimri]

              Implementation of Exactly Once in Order (EOIO) scenario in Integration Suite

Introduction:

This blog post demonstrates how to achieve exactly once in order (EOIO) functionality within cloud Integration. As of now there is not out of box functionality provided in Integration suite for EOIO scenarios. We need to develop custom integration flows to achieve the requirement where sequence needs to be maintained. SAP eventually plans to introduce EOIO functionality in Integration Suite by Q1 of 2025 as per below roadmap.

https://roadmaps.sap.com/board?PRODUCT=40F2E9281A631EDA94C7CBA39DC9E0F4&range=CURRENT-LAST#;INNO=252...

Usecase: Process files in sequential based on file name(ex: sequence#) and send the payload to target application preserving the order (EOIO).

Prerequisite:

We need a staging sftp server to temporary store the file before sending to Target application in sequence. We are utilizing the sftp poll enrich property ‘SAP_PollEnrichMessageFound’ to implement the solution as explained in below sections. Also, for simplicity we are using the file number in the file name (e.g. TestEOIO_<filenumber>.txt) to determine the sequence in which files are delivered to the Target system.

Solution: 

High level solution is to use two data stores. One for storing key field values and another one for actual payload.

1st IFlow: Get the files from staging SFTP Server folder, store the payload(xml), key(xml) in two data stores respectively with the same entry id.

2nd IFlow: Read data store (keys), fetch, sort and split the keys (xmls). For each split key(xml), read the corresponding payload(xml) from 2nd data store and send it to target application. 

Create two integration flows as below.

1st Integration Flow

• Get the file from staging SFTP folder using poll enrich to pick one by one in any order. Use Poll enrich property SAP_PollEnrichMessageFound to determine if there is any file left to be picked from SFTP staging folder at a given point of time.

• If ${property. SAP_PollEnrichMessageFound}=’true’ keep picking next available file. If no file exists, then call the second integration flow using ProcessDirect call.
• Create a key which is used to determine the sequence of file delivery to target. In our case we used File sequence number in the end of the file name.

• Once the key is extracted, store the key as an xml payload in data store (DS_TEST_EOIO_KEY)

Also store the actual file payload in a separate datastore using the same Key value.

Once all the keys and actual file payloads are stored in their respective datastores the 1st iflow job is done.

2nd Integration flow

The second iflow is used to fetch all the key xml’s from the datastore (DS_TEST_EOIO_KEY) and sort the Key values. Then based on sorted Key values fetch the corresponding Payload from datastore (DS_TEST_EOIO_PAYLOAD). This integration flow is triggered only when the there is no more file to pick from source/staging folder in the first flow by using SAP_PollEnrichMessageFound property.

• Select the all the available Key xml data from Datastore (DS_TEST_EOIO_KEY). Provide a maximum number like 9999 in Number of polled messages field. This will help to select all the available Keys from the datastore.

• Sort the fetched Key xml records and split them.

• Extract the sorted key.

• Get the payload based on sorted key.

• Once the message is sent successfully to target using Send step then only delete the entry from both the data stores. This is important to maintain the transaction handling if any error occurs during file transmission to Target or any mapping/validation error due to data issue.

Result

In our test scenario we are receiving the files to a target SFTP server without disrupting the sequence.

Source folder can get the file in random order.

Target Server received the files in sequence.