How to achieve EOIO using SFTP Adapter in SAP Integration Suite

[Jayakumar-V]

Introduction: 

When dealing with file transfers through the SFTP adapter in SAP CPI, the challenge of ensuring that each file is processed one at a time in order is pivotal. Currently SAP CPI doesn't have a standalone feature of EOIO processing as in SAP PO.

In this blog, I've explain how to achieve "Exactly Once In Order" processing within the SFTP adapter of SAP CPI.

Business requirement:

To transfer files from external SFTP server to SAP systems or to external applications.

The file should be processed exactly once in order. After successful completion of previous file only the next file should be processed by SAP CPI.

Challenges:

Exactly once in order processing is difficult to achieve in SAP CPI using the standard features, since there was no build in feature in SFTP adapter.

In case, the client CPI tenant has multiple runtime nodes, SFTP adapter will pick each file per node in a single poll, even if the "Max message per poll" parameter is set as 1.

Let's explore the step-by-step process of how to achieve this requirement by creating an integration flow using SFTP adapter in SAP CPI.

Pre requisites: 

The filename in the SFTP server should be static and have a sequence number at the end. Ex: abcd_1.txt

A done file is expected to trigger the sequential processing flow. The .done file should be under the same directory as the original files.

Setting up an Integration flow:

Let's create an integration flow with an SFTP sender adapter.

SFTP Sender Adapter configurations:

In the processing tab, select the Read Lock Strategy as 'Done File Expected'. This will by default takes the source filename with .done extension. We can rename it as per our requirements.

Select Sorting Order as 'Ascending' and Sorting as 'Timestamp', to follow the first in first out.

Set Maximum message per poll as '1' (This parameter is optional in our case)

Now, the sender adapter will poll for the respective file, if the file found in the directory it'll additionally check for the done file with same name. Ex: If the file name was 'Test_1.xml', it'll check for 'Test_1.xml.done' file to send the message.

After processing the message, the integration flow will create a .done file for the next file in the sequence.

Let's use a groovy script to create the next file name in the sequence.

import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
def Message processData(Message message) {

//get Headers
    def headers = message.getHeaders();
    def value = headers.get("CamelFileName");

//get & remove the extension from file name
    def extension = value.substring(value.lastIndexOf('.'),value.length())
    def withoutExt = value.replaceFirst(/\.[^\.]+$/, '')

//get & remove the sequence number at end
    def count = withoutExt.split("_")[-1]
    count++
    def nameWithoutCount = withoutExt.replaceFirst(/\_[^\_]+$/, '')
   
//create filename with next sequence number
    def FileName = nameWithoutCount+"_"+count+extension+".done"
    message.setProperty("FileName", FileName);
    return message;
}

This script will create the .done file for the next file in the sequence. This .done file acts as a flag to process the next file from the SFTP server.

For example: If the first file processed has a filename of 'Test_1.txt' the script will create the filename as 'Test_2.txt.done'. 

SFTP Receiver adapter configurations:

All configurations should be same as the sender - Directory, Address, Credentials. For File Name, use the filename created using the script.

Let's test our scenario:

Files are placed in the server, with a .done file to trigger the process

After successful processing of Test_1.txt, the .done file for next sequence will be created in the server. 

This will create a streamline process and transfer files one by one in sequence by SAP CPI.

Thanks for reading the blog !! Happy Integrating.