General Data Splitter Tool for Optimized Efficient Data Processing and Scalable Integration

[ManaswiniPuli]

Struggling with handling massive data files in your integration flows? You're not alone! I'm Manaswini Puli , committed to break down complex integration challenges. In this hands-on guide, I'll walk you through using the General Splitter tool in SAP Integration Suite to efficiently break down large messages into manageable segments—boosting processing speed and scalability while handling structured XML/JSON data with ease.

Follow along as we transform complex data streams into optimized, scalable workflows step by step. If this solves your data bottleneck challenges, I'd love to hear your thoughts in the comments.

Below is the complete iFlow, featuring the use of a General Splitter to process structured messages effectively, along with OData integration for seamless data exchange.

Step-by-Step Guide: General Data Splitter Tool for Optimized, Efficient Data Processing and Scalable Integration

Step 1: Access Integration Suite

1. Log in to SAP BTP Cockpit using your registered email and password.
2. Go to your Subaccount and navigate to Subscriptions.
3. Click on Integration Suite.

Step 2: Create a Package

1. In Integration Suite, go to the Design tab and click Packages.
2. Click Create Package, enter a valid name and description, then click Create.

Step 3: Create an Integration Flow

1. Open the package you created.
2. Click Create, select Integration Flow, enter a valid name and description, then click Create to open the editor.

Step 4: Configure HTTPS Sender Adapter

1. Add an HTTPS sender to receive incoming data.
2. Under the Connection tab:
• Set the Address Path to /Products/Details.
• Set Authorization to User Role.
• Enter User Role: ESBMessaging.send.

Reason:

• The Address Path defines the endpoint where the integration flow listens for incoming HTTPS requests, allowing external systems to send product data.
• The ESBMessaging.send role ensures only authorized clients can access the endpoint.

Step 5: Add General Splitter

      1. Add  a General Splitter step after the HTTPS sender.

     2.  Under the Processing tab:

• Set Expression Type to XPath.
• Reason:The payload is in XML format, so setting the expression type to XPath enables precise navigation and extraction of relevant data elements.
• Set XPath Expression to //productIdentifier.
• Reason: productIdentifier is the key element in the payload that uniquely identifies each product, enabling extraction and individual processing.
• Set Grouping to 1.
• Grouping controls how many split messages are processed together.
• Reason: The payload will be taking 3 product Identifiers, so the messages will be divided into 3 groups, each containing a single product Identifier for individual processing.

Step 6: Add Content Modifier

1. Add a Content Modifier step after the HTTPS sender.
2. Go to the Message Header section and click Add.
3. Configure the header as follows:
• Name: productIdentifier
• Source Type: XPath
• Source Value: //productIdentifier
• Data Type: java.lang.String
4. Reason: This extracts the productIdentifier value from the XML payload and stores it as a string in the message header for use in subsequent processing steps.

Step 7: Add OData Request-Reply

1. Add an OData Request-Reply step after the General Splitter.
2. In the Connection tab, configure the following:

Address:https://refapp-espm-ui-cf.cfapps.eu10.hana.ondemand.com/espm-cloud-web/espm.svc

Reason: This OData endpoint is used to retrieve product data based on the productIdentifier stored in the message header.

          3.In the Processing tab, configure the following:

• Set Operation Details to Query (GET) to retrieve data.
  Reason: Using the GET query operation allows fetching specific data from the OData service efficiently based on defined query parameters.
• Click Select under Resource Path to specify the data source:

       Step 1: Connect to System

    Choose Remote as the connection source to connect to the external OData service.

     Step 2: Select Entity & Define Operation

    From the dropdown list, select the entity you want to query (for example, Products).
    After selecting the entity, specify the fields to optimize the data returned by the OData service.

       Step 3: Configure Filter & Sorting

    Under Filter By, set the filter condition:
    productId equals ${header.productIdentifier}

• Click Finish to complete the configuration.

Step 8: Add Gather

1. Add a Gather step at the end of the flow, after the OData Request-Reply.
2. In the Processing tab, configure:
• Incoming Format: XML
• Aggregation Algorithm: Combine
  Reason: The input messages are in XML format, and the Combine algorithm joins them into one XML message. This helps gather all individual responses into a single output.

Step 9: Deploy and Test Integration Flow

1. Deploy your integration flow from the editor.
2. After deployment, get the endpoint URL generated for the HTTPS sender adapter.
3. Open Postman to test the integration flow.
4. Set the HTTP method to POST.
5. Use the endpoint URL as the request URL.
6. Under Authorization, select OAuth 2.0 and provide the following details from the service key created for the integration flow:
• Client ID
• Client Secret
• Token URL

     7.In the Body tab, provide the XML payload or data you want to send.

<?xml version='1.0' encoding='UTF-8'?>

<root>

  <record>

    <Products>

      <productIdentifier>HT-2000</productIdentifier>

      <productIdentifier>HT-1138</productIdentifier>

      <productIdentifier>HT-1137</productIdentifier

    </Products>

  </record>

</root>

        8.Send the request and verify the response.

Step 10: Check the Processed Output

• The input payload includes 3 productIdentifier entries.
• With the General Splitter configured to group by 1, the message is split into 3 individual groups, each containing 1 product.
• The following is Group 1, containing 1 productIdentifier as a result of the splitter configuration.

• The following is  the OData response, which returns the product details for that specific productIdentifier.

• At the Gather step, these messages are reassembled into a single message that contains all 3 processed product entries.

Disclaimer:
All the views and opinions in this blog are my own and are made in my personal capacity. SAP shall not be responsible or liable for any of the contents published in this blog.