- SAP Managed Tags:
In this blog post, you will learn how to better model a Content Enrich scenario in SAP Cloud Integration. All full-scale integrations would inevitably have one or more Enrich steps. Very often, the adapter connected to the Content Enricher pulls in all the available data from the backend system causing crashes and performance degradation. The efficiency of the integration therefore greatly depends on the finer modeling of the Enricher step especially when large data is involved.
In this blog post, we will model a simple integration flow to handling large data is using dynamic queries using property and OData v2 $filter query option. In the example here, we will be enriching CompoundEmployee entity with FOLocation by creating a dynamic filter in Successfactors OData v2 adapter.
I have configured a looping process(as seen below) to fetch and enrich Compound Employee records in pages. The script step "Parse keys to Properties" has the logic for creating a dynamic filter.
Content Enricher configuration for this scenario is shown below.
Content Enrich works by matching the key elements from Original Message(i.e. Compound Employee) to the key elements in Lookup Message(i.e. FOLocation) and then intelligently aggregating the snippets of the matching data from Lookup into the Original message.
We can always help optimize the data fetched in the Lookup Message by passing the appropriate key elements as $filter parameter for the adapter. In this example, we are comparing key element location from Compound Employee to key element externalCode and therefore we will define and configure a dynamic filter on externalCode for optimization.
The main step. Below is the Script Code for "Parse keys to Properties" to construct the dynamic filter. The first part of the script parses the Original Message and selects all the key elements using XmlSlurper. The second part forms the key element in the OData $filter format and sets it to property named leadKey.
You can see that I always have the property configured with a value. Else there would be a runtime failure if the property is missing.
PS: "in" operator is unique to Successfactors. For standard OData services ( such as Hybris Marketing, XSOData etc), you will need to use the "or" operator instead.
The simplest step. Add the property leadKey in the query option. And your scenario is good to go!
The above steps might look a bit overwhelming but is relatively easy to perform. The script I shared must suit your use case with minimum changes. In general, dynamic parameters can greatly help improve the performance of your integration scenarios and most adapters support dynamic injection of parameters. Do consider them in all your integration flow designs.
In this blog post, we will model a simple integration flow to handling large data is using dynamic queries using property and OData v2 $filter query option. In the example here, we will be enriching CompoundEmployee entity with FOLocation by creating a dynamic filter in Successfactors OData v2 adapter.
The Scenario
I have configured a looping process(as seen below) to fetch and enrich Compound Employee records in pages. The script step "Parse keys to Properties" has the logic for creating a dynamic filter.
Content Enricher configuration for this scenario is shown below.
Understanding Content Enrich Functionality
Content Enrich works by matching the key elements from Original Message(i.e. Compound Employee) to the key elements in Lookup Message(i.e. FOLocation) and then intelligently aggregating the snippets of the matching data from Lookup into the Original message.
We can always help optimize the data fetched in the Lookup Message by passing the appropriate key elements as $filter parameter for the adapter. In this example, we are comparing key element location from Compound Employee to key element externalCode and therefore we will define and configure a dynamic filter on externalCode for optimization.
Configuring Content Enrich to work with Large Data Volumes
Step 1: Defining the script
The main step. Below is the Script Code for "Parse keys to Properties" to construct the dynamic filter. The first part of the script parses the Original Message and selects all the key elements using XmlSlurper. The second part forms the key element in the OData $filter format and sets it to property named leadKey.
You can see that I always have the property configured with a value. Else there would be a runtime failure if the property is missing.
PS: "in" operator is unique to Successfactors. For standard OData services ( such as Hybris Marketing, XSOData etc), you will need to use the "or" operator instead.
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
def Message processData(Message message) {
//Body
def body = message.getBody(java.io.Reader);
//Parse the key from Original message
def response = new XmlSlurper().parse(body);
def keys = response.'**'.findAll{ node-> node.name() == 'location' }*.text();
//Formulate the filter query. Eg: $filter=externalCode in '1710-2018'
def leadKey = "'" + keys.unique().join("','") + "'";
if(leadKey == null || leadKey.trim().isEmpty()){
message.setProperty("leadKey", "");
} else{
message.setProperty("leadKey", "\$filter=externalCode in " + leadKey);
}
return message;
}Step 2: Configure the SucessFactors OData v2 adapter
The simplest step. Add the property leadKey in the query option. And your scenario is good to go!
Advantages of the approach
- Far lesser calls to the backend systems.
- Lesser and importantly relevant data only pulled in each enrich call.
- Overall improvement in throughput and execution times.
Points to consider
- Only elements defined as filterable can be used in a $filter query. A good design approach could be to select only fields that are filterable as Lookup key elements. In case the required key element is not filterable, you can also consider another filterable element that is related to the selected key element to form the query.
<Property Name="externalCode" Type="Edm.String" Nullable="false" sap:filterable="true" sap:required="true" sap:creatable="false" sap:updatable="false" sap:upsertable="true" sap:visible="true" sap:sortable="true" MaxLength="32" sap:label="Code"/>- API also can have limit restrictions. For instance, the 'in' operator in Successfactors only supports 1000 values in a call. You may need to have a looping process to handle these cases.
- Adding dynamic queries increases your overall query length and can lead to issues. The server usually has a limit on allowed length for a query. This is defined typically in power of 2, i.e. 4096(2^12) or 8192(2^13) with Successfactors.
- Process in pages option does not work with Enricher.
- Leaving the page size empty would be the best for most endpoints. This would ensure all the records are fetched optimally based on server paging configuration.
The above steps might look a bit overwhelming but is relatively easy to perform. The script I shared must suit your use case with minimum changes. In general, dynamic parameters can greatly help improve the performance of your integration scenarios and most adapters support dynamic injection of parameters. Do consider them in all your integration flow designs.
12 Comments
