SAP Cloud Integration: Read and Write files using AmazonWebServices S3 Adapter

[former_member767559]

From January 2021, a new set of OEM adapters were launched. Amazon Web Services was one of them. This adapter supports several protocols including S3, SQS, SNS, and SWF. A high-level overview of the adapter was shared in this blog post.

In this article, we will walk you through the AWS S3 sender and receiver adapter using a simple scenario.

Assuming that a third-party system is producing files and placing them in an S3 Bucket. You have been asked to pick these files and place the files in another S3 bucket. The scenario is depicted in the Figure below:

Figure 1: Scenario

Your Integration Flow in SAP Cloud Integration

Figure 2: Integration Flow

Step 1: Select and Configure AmazonWebServices Sender Adapter:

The first step is to use the AmazonWebServices adapter and select S3 as the protocol to pick up the incoming files from the S3 bucket. In this scenario, an S3 bucket named “ci-demo” is used. The configuration of the sender adapter and connection tab can be seen below:

Figure 3: Sender Adapter Connection Tab

Different parameters of the connection tab are explained in the below table:

Parameter name	Description
Region Name	AWS region where the Bucket reside.
Bucket Name	Name of the bucket on AWS where the Sender Adapter should poll files from.
Polling Interval (in ms)	Polling Interval in milliseconds. Default value: 60000
Access Key Alias	Name of the secured parameter which stores AWS Access Key.
Secret Key Alias	Name of the secured parameter which stores AWS Secret Key.

Table 1: Sender Adapter Connection Tab Parameters

Furthermore, the Figure below depicts the configuration of the Processing tab.

Figure 4: Sender Adapter Processing Tab

Some of the parameters of the processing Tab are explained in the below table:

Parameter name	Description
Directory	The path on the AWS bucket from where the file should be read. Example: "ci/in" The value can also be read dynamically using a header or property. If left blank, then the adapter will read the root directory.
File Name	The filename is to be read. Example: testfile.txt Other Supported values: Wild Card Characters, an asterisk (*), and question mark (?). When left blank, the adapter will read all the files present in the S3 directory.
Sorting	The property on which sorting should be done while polling files. The sorting will be done in ascending order of the option selected. In our scenario, we are not using this option but below are the available options: File Name: The sorting is performed based on the filename. File Size: Sorting is performed based on file size. Time Stamp: Sorting is performed based on the timestamp.
Include Sub-Directories	Select the checkbox to read all the files in the directory and subdirectory.
Post-Processing	Specify the action that should be taken after the file has been processed by the adapter. Available options: Delete File: Select this option to delete the file after processing. This is the default option for Post-Processing. Keep File and Process Again: Select this option to keep the file at the source directory and process again. Move File to an Archive directory: Select this option to move the file to an archive directory. After selecting this option, specify the archive directory. You may also choose to add a timestamp in the archived file. Copy file to an Archive Bucket: Select this option to copy the file to another bucket. After selecting this option, specify the archive bucket and archive directory. You may also choose to add a timestamp in the archived file. Move file to an Archive Bucket: Select this option to move the file to another bucket. After selecting this option, specify the archive bucket and archive directory. You may also choose to add a timestamp in the archived file.
Optional S3 Metadata	Specify the metadata to be populated from S3 object metadata. Multiple metadata can be specified separated by comma (,). If specified, then a property with the same name will be created in Cloud Integration. Example: If you assume that the file in the S3 bucket has metadata named “MyMetadata” associated with it. Then to retrieve the metadata in the iFlow, “MyMetadata” has to be specified in the adapter. During execution, a property with the name “MyMetadata” will be created in Cloud Integration and can be accessed using the expression  ${property.MyMetadata}.

Table 2: Sender Adapter Processing Tab Parameters

Step 2: Prepare filename for the file on the target side:

This step is optional. This is just to showcase that the filename can be extracted and later used to dynamically set the filename in the receiver adapter from any property or header created during the execution.

In this case, we have created a property called “filename” which we will use in the next steps while creating the file on the receiver side. See below:

Figure 5: Content Modifier for Filename

Step 3: Select and configure AmazonWebServices receiver adapter

Use the AmazonWebServices receiver adapter with S3 protocol to create a file on the S3 bucket. The configuration of the adapter (Connection Tab) is shown in the below picture.

Figure 6: Receiver Adapter Connection Tab

Details of the meaning of the properties in the adapter are shown below:

Parameter name	Description
Region Name	AWS region where the Bucket reside. Select the region from the pre-configured list. If the region does not exist in the pre-configured list, then the user can manually enter the relevant region name.
Bucket Name	Name of the bucket on AWS where the Receiver Adapter should place the file. Example: cloud-integration-demo
Access Key Alias	Name of the secured parameter which stores AWS Access Key. The values can be read dynamically.
Secret Key Alias	Name of the secured parameter which stores AWS Secret Key. The values can be read dynamically.

Table 3:Receiver Adapter Connection Tab Parameters

The figure below shows an example configuration of the Processing Tab.

Figure 7: Receiver Adapter Processing Tab

Note that the receiver S3 adapter supports two different operations:

Write operation: This operation is used to write files to the Amazon S3 bucket.

Read operation: This operation is used to read a single file from the Amazon S3 bucket in the middle of the process. This will be covered in the upcoming blog post.

 

Parameter name	Description
Directory	Specify the location where the file should be written. Example: "ci/out" The value can also be read dynamically using a header or property​.
File Name	Specify the name of the file to be written. Example: "filename.txt". The value can also be read dynamically​ using a header or property.
Content Type	Specify the file's MIME type. Example: "application/xml" or "text/plain". The value can also be read dynamically using a header or property​.
Storage Class	Select the AWS Storage Class. Example: "Standard". You can choose between the following available options: ·        Glacier  ·        Glacier Deep Archive  ·        Intelligent-Tiering  ·        One Zone-Infrequent Access  ·        Reduced Redundancy  ·        Standard  ·        Standard-Infrequent Access
Existing File Handling	Select the file handling behavior if a file with the same name exists in the directory. Example: "Overwrite". You can choose between the following available options: ·        Fail: Select this option to raise an exception.  ·        Ignore: Select this option to ignore the file.  ·        Overwrite: Select this option to overwrite the existing file.
Empty File Handling	Select the file handling behavior if the file is empty. Example: “Proceed with Empty File” You can choose between the following available options: ·        Proceed With Empty File: Select this option to create an empty file. ·        Ignore: Select this option to ignore the empty file.

Table 4: Receiver Adapter Processing Tab Parameters

Step 4: Execution and Monitoring

Now we will go ahead and place a file in the source S3 bucket.

The Iflow polls the above folder at the defined polling interval in the sender adapter. After te  file has been successfully picked up by the IFlow, an execution entry can be seen the Monitoring in the SAP Integration Suite. See below.

Let's go and look at the traces after the file has been picked up by the sender adapter. The below-listed exchange properties are created by default every time a file is picked up by the sender adapter.

You can use these values at any subsequent steps in your Iflow by using the general expression for the properties. Example: ${property.S3ObjectName}.

We could see that the file has been created in the target directory. The filename is appended with a timestamp as selected in the receiver adapter.

Congratulations. You have now learned about the S3 protocol of the Amazon Web Services adapter. Watch out for the upcoming blog post on Amazon Web Services adapter.