We have recently shipped a new version 1.0.8 of the Pipeline Concept were we introduced the following new features:
- Option to combine receiver and interface determination in one single XSLT mapping for an improved runtime behavior
- Option to bypass interface determination pipeline step in case of Recipient List pattern without interface split
- Option to bypass receiver determination pipeline step in case of interface split scenarios with one single receiver only
- Supporting header-based XSLT routing conditions if message body is non-XML
- Extension for custom receiver determination instead of XSLT from Partner Directory
- Extension for custom interface determination instead of XSLT from Partner Directory
- New pipeline processing log providing more details if message is stored to the dead letter queue
- Fixed standard retry handling for generic integration flows with splitter to avoid message duplicates
The package can be accessed from the SAP Business Accelerator Hub. If you have already copied the package to your workspace in your SAP Integration Suite tenant, you can simply run the update of the package to be able to use the latest features.
Let me briefly describe some of the new features.
Further bypass options
With version 1.0.6, we have already shipped a bypass option for pure Point-to-Point scenarios, see New Pipeline Concept features: new partner id definition, alternative partner, bypass option. With the latest version, you now have the option to either bypass the receiver determination or the interface determination depending on your integration scenario. For an interface split scenario with one single receiver only, you can bypass the receiver determination. For a Recipient List pattern with one interface per receiver, you can bypass the interface determination. In any cases, by applying the different bypass options you can save at least one JMS queue and one pipeline step which leads to an improved runtime behavior.
Further runtime improvements can be achieved by combining the receiver determination xpaths and the interface determination xpaths in one single XSLT. This use case is actually a special case of the interface determination bypass option. Here, the receiver determination pipeline step has been enhanced supporting a receiver determination XSLT that contains xpaths conditions for both the receiver and the interface determination. The receiver determination pipeline step then splits both the number of receivers and the number of interfaces. This way, you only need to read, decode, and run one single XSLT instead of multiple interface determination XSLTs for each receiver.
Here's an example of such a combined XSLT:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:param name="dc_country"/>
<xsl:template match="/">
<ns0:Receivers xmlns:ns0="http://sap.com/xi/XI/System">
<ReceiverNotDetermined>
<Type>Error</Type>
<DefaultReceiver/>
</ReceiverNotDetermined>
<xsl:if test="/*:Item/Category = 'Keyboards'">
<Receiver>
<Service>Receiver_11</Service>
<Interfaces>
<xsl:if test="/*:Item/Quantity = '10'">
<Interface>
<Index>1</Index>
<Service>/PIP/Step07/Samples/Scenario_1/Receiver_11/IfIdx1</Service>
</Interface>
</xsl:if>
<xsl:if test="/*:Item/CurrencyCode = 'EUR'">
<Interface>
<Index>2</Index>
<Service>/PIP/Step07/Samples/Scenario_1/Receiver_11/IfIdx2</Service>
</Interface>
</xsl:if>
<xsl:if test="/*:Item/Quantity != '10'">
<Interface>
<Index>3</Index>
<Service>/PIP/Step07/Samples/Scenario_1/Receiver_11/IfIdx3</Service>
</Interface>
</xsl:if>
</Interfaces>
</Receiver>
</xsl:if>
<xsl:if test="(/*:Item/Category = 'Keyboards') or (/*:Item/Category = 'Software')">
<Receiver>
<Service>Receiver_12</Service>
<Interfaces>
<xsl:if test="$dc_country = 'DE'">
<Interface>
<Index>1</Index>
<Service>/PIP/Step07/Samples/Scenario_1/Receiver_12/IfIdx1</Service>
</Interface>
</xsl:if>
<xsl:if test="$dc_country != 'DE'">
<Interface>
<Index>2</Index>
<Service>/PIP/Step07/Samples/Scenario_1/Receiver_12/IfIdx2</Service>
</Interface>
</xsl:if>
</Interfaces>
</Receiver>
</xsl:if>
</ns0:Receivers>
</xsl:template>
</xsl:stylesheet>
As you can see, the XSLT contains the receiver components as well as the receiver interface index and the end points of the scenario-specific outbound processing flows where the mappings and the delivery to the receivers are done. By providing the end point information here, the interface determination pipeline step is bypassed.
For more details, see Pipeline Bypass Options.
Custom extensions
As an alternative to the standard receiver or interface determination based on an XSLT from the Partner Directory, you can implement your own custom receiver or interface determination. The generic integration flows support a kind of custom exit that you can use to run your own custom logic without changing the delivered generic integration flows. To do so, you must implement a separate scenario-specific integration flow that's called instead of the standard XSLT mappings.
You do not need to create the scenario-specific integration flows from scratch, the updated Pipeline package contains templates that you can copy and adapt to your needs. The templates have the target XSDs as references which eases the implementation effort. So, you can create a message mapping that maps to the target structure or alternatively you can create your own groovy scripts or use any other logic which you prefer. The target XSD for the receiver determination by the way supports the combined receiver and interface determination as described above. So, when using the custom extensions, you can also leverage the bypass options.
For more details, see Customizing the Pipeline Concept.
Non-XML payloads
When running an XSLT mapping to determine the list of receivers or interfaces for non-XML messages, you usually run into an error even if the conditions are based on headers only because the XSLT expects an XML. To overcome this error, a dummy XML is created right before running the XSLT. With this, you are able to process non-XML messages within the pipeline without the need to convert into XML before.
Processing log
At various places during message processing, information is stored in a new header which we pass across the pipeline steps. We decided to add the log information whenever we call groovy scripts to avoid to make too many changes to the integration flow model. Whenever a message is stored in the dead letter queue because the maximum of retries has been exceeded, the log information is attached to the message processing log.
Retry handling
For the generic receiver and interface determination integration flows where the message is split, transaction handling ensures that split messages are rolled back in case of an error. However, if the message is forwarded to the dead letter queue, the message processing ends with either a message end event or an escalation end event so that in rare cases already split messages might be committed. This may lead to duplicates if the message is moved from the dead letter queue to the actual queue for reprocessing. To avoid duplicates, we had to change the standard retry handling within the receiver and the interface determination flows. When the actual maximum number of retries have been exceeded, we need to do an additional retry and then check right at the beginning of the integration flows before splitting the message. For the other generic flows that do not carry out a split, the retry handling kept unchanged.
If you like to try out the pipeline concept in general and the new features in particular, check out this github repository where I describe how to setup sample scenarios using the pipeline concept. Here, we have added the following new scenarios and scenario variants leveraging the new features:
As usually, in case of feedback, feel free to reach out to me.
