New Pipeline Concept features: integrated runtimes...

alex_bundschuh · ‎2025 Jan 29

I like to let you know that a new version 1.0.9 of the Pipeline Concept has been recently shipped with a couple of new enhancements that help you to improve the overall performance, operation and lifecycle when using the Pipelines package for Cloud Integration. At a glance, the following new features and fixes have been introduced:

Supporting different tenant stages, e.g., DEV, QA, PRD
New integrated messaging runtime "Pipeline Generic Step02 - Integrated Messaging Runtime Async" combining inbound conversion and receiver/interface determination in one pipeline step
New integrated messaging runtime "Pipeline Generic Step02 - Integrated Messaging Runtime Sync" for handling synchronous Content-Based-Routing scenarios
New template "Pipeline Template Step01 - Inbound Processing Synchronous" for scenario-specific synchronous inbound processing
New template "Pipeline Template Step07 - Outbound Processing Synchronous" for scenario-specific synchronous outbound processing
Generic inbound processing for XI supports both asynchronous and synchronous messaging
Failed test messages end with escalation instead of completed
Added custom error handling for generic XI and IDoc inbound flows

The package can be accessed from the SAP Business Accelerator Hub. If you have already copied the package to your workspace in your SAP Integration Suite tenant, you can simply run the update of the package to be able to use the latest features.

Let me briefly describe the major new features.

Integrated messaging runtime - Async

In the previous package version 1.0.8, we shipped the option to combine receiver and interface determination in one single XSLT mapping for an improved runtime behavior. Now, with the new version, we introduced a new pipeline step called Pipeline Generic Step02 - Integrated Messaging Runtime Async which combines the inbound conversion as well as the receiver and interface determination in one single pipeline step. This helps to further improve the runtime behavior as well as monitoring and operation.

When using this new integrated pipeline step, you are able to skip most of the other pipeline steps. So, you can reduce the number of JMS queues required to process your messages from 4 to 2 or 8 to 4 when counting in the dead letter queues as well. This integrated messaging runtime only works with the combined receiver and interface determination XSLT, so we can reduce the number of XSLT to a minimum of 1. If you like to use this integrated pipeline instead of the fully decoupled pipeline, you just need to deploy the new generic flow and configure the JMS queue of the scenario-specific inbound processing flow accordingly, by default the name of the inbound processing queue of the integrated pipeline is PIPX01 instead of PIPQ01.

From performance measurements that we did, we know that reading and writing from and to the JMS queues incl. zipping and unzipping the messages as well as fetching the XSLT mappings from the Partner Directory incl. decoding the XSLT require the most CPU consumption within the overall pipeline processing. So, using the integrated messaging runtime helps to reduce the overall CPU consumption compared to the fully decoupled pipeline with 4 generic integration flows instead of 2.

Furthermore, since we run through less number of pipeline steps, the number of message processing logs is also reduced.

We still support the complete pipeline which is shown here:

Recommendation however is to use the integrated pipeline. This picture nicely illustrates which steps and queues can be saved.

For more details about the integrated messaging runtime, see the chapter Integrated Messaging Runtime (Asynchronous) in the standard documentation.

Integrated messaging runtime - Sync

With the new version, we now also support a synchronous pipeline. The new pipeline called Pipeline Generic Step02 - Integrated Messaging Runtime Sync combines the receiver and interface determination in one single pipeline step and is suited for handling synchronous Content-Based-Router scenarios. Other than the async pipelines, it doesn't require any JMS queue, instead the pipeline steps are connected using the ProcessDirect adapter which is synchronous by nature.

For more details about the synchronous integrated messaging runtime, see Integrated Messaging Runtime (Synchronous).

Supporting landscape stages

On Cloud Integration, you do not necessarily have to have different business system names depending on the landscape stage such as DEV, QA, PRD. For XI and IDoc however you usually do. In SAP Process Orchestration, you maintain the business systems as well as transport groups in its SLD, and when transporting the objects between the stages, the business names are automatically converted. This is not supported in Cloud Integration, so we came up with the following approach to support different stages without the need converting the system names when transporting across the stages.

Within the pipeline, we actually use alias names for the business systems. During runtime, the actual sender business system name is mapped to a sender alias which is used within the pipeline processing, once the receiver alias has been determined, it is then mapped to the actual receiver business name. This allows you to run your scenarios on the different landscape stages without changing the configuration in the Partner Directory such as the XSLT for determining the receivers. So, all partner and scenario definitions in the Partner Directory are the same on all stages based on the alias names. As said, there is no need to do any changes in the XSLT, so this approach is less error prone.

To setup the support for landscape stages, you need to create the business system aliases as partner IDs in the Partner Directory where for each stage you maintain the mapping between alias name and actual names.

Furthermore, to automatically identify the landscape stage of your tenant during runtime, the tenant names are used which are mapped to the corresponding stage IDs. This mapping is stored in the Partner Directory using partner ID SAP_Integration_Suite_Landscape. To do so, you can either use the Partner Directory API or the new Partner Directory user interface.

For more details, see Landscape Stages.

Single point of entry for any XI inbound scenarios

With the latest enhancements of the Cloud Integration capability of SAP Integration Suite, we support the so called handled by integration flow option for both the XI sender and the XI receiver adapter. In the generic XI inbound processing flow we have changed the XI sender adapter using this very option. With this, we can reduce the number of JMS queues by one. Since we write the messages into an inbound JMS queue anyway, we do not need to use the XI sender specific JMS queue. Furthermore, this option allows us to receive both async and sync calls via the same inbound adapter. So, this reduces the configuration effort in your sender backend, there is no need any more to maintain the IS_URL configuration parameter, all scenarios can be pointed to the very same end point on Cloud Integration.

For more details, see XI Sender Adapter.

If you like to try out the pipeline concept in general and the new features in particular, check out this github repository where I describe how to setup sample scenarios using the pipeline concept. Here, we have added the following new scenarios and scenario variants leveraging the new features:

Scenario 2D uses the new asynchronous integrated messaging runtime for a recipient list and interface split scenario.
Scenario 8 handles a synchronous Point-to-Point with XI inbound.
Scenario 9 uses the new synchronous integrated messaging runtime for a synchronous content-based-router pattern.

As usually, in case of feedback, feel free to reach out to me.

Patrick_Schaaf · ‎2025 Jan 30

Where can the partner directory user interface be found?

In SAP help, still only the API option is listed:

Partner Directory | SAP Help Portal

mate_moricz · ‎2025 Jan 30

Hi Patrick,

You can find in the Monitoring landing page under the "Manage Stores" section:

https://help.sap.com/docs/integration-suite/sap-integration-suite/managing-partner-directory-entries

Regards,

Mate

MatthiasLechner · ‎2025 Mar 17

Hi @alex_bundschuh,

when determining the partner_id for the XI part (ABAP proxy), is it intended to not take the namespace into consideration? Sender System and Interface is not unique, as we can have multiple interfaces with the same name in different namespaces. Or have I overseen something?

Thanks and BR,

Matthias

alex_bundschuh · ‎2025 Mar 19

@MatthiasLechner so far, we do not support namespaces. We will look into this.

Alex

Benko · ‎2025 Mar 26

"Generic inbound processing for XI supports both asynchronous and synchronous messaging"

Has anyone got this to work for Synchronous proxies?

My scenario works perfectly fine with:

Pipeline Template Step01 - Inbound Processing Synchronous
Pipeline Generic Step02 - Integrated Messaging Runtime Async
Pipeline Template Step07 - Outbound Processing Synchronous

but the same message with this configuration doesn't work:

Pipeline Generic Step01 - Inbound Processing for XI
Pipeline Generic Step02 - Integrated Messaging Runtime Async
Pipeline Template Step07 - Outbound Processing Synchronous

ERP reports an empty response payload, even though the response message is visible in all the traces.
"Pipeline Generic Step01 - Inbound Processing for XI" is left at status "Processing" forever.

Regards

Steve

alex_bundschuh · ‎2025 Mar 27

@Benko this is a bug of the handled by integration flow feature, I have reported an internal incident with high prio, development are working on fixing it and will patch the tenants soon, as long as this hasn't been fixed you may revert to version 1.0.2 of the generic XI inbound flow, version 1.0.2 however would then only work for async scenarios

Alex

ronald_fassl1 · ‎2025 Apr 07

We activated the integrated messaging runtime and use it for a high load use case (>100.000 proxy messages per day).

We have used this scenario with PIP configuration already before without integrated runtime and everything went smooth. With the usage of the integrated runtime we receive for 1 out of 10.000 messages an error in the PIP07 step (scenario specific outbound).

The error says something about an invalid xpath expression, but that's not the important thing.
The problem is, that there is absolutely no payload attached to this message (and therefore the xpath validation fails). When I go in retry mode from PIP04_DLQ to PIP04 and activated the trace for the corresponding PIP07 iFlow, there is no payload attached.
The source message is an SAP backend proxy message and I can verify that the proxy message was sent with a valid payload and it was not empty.

We haven't seen this behaviour in the weeks before without the usage of integrated runtime, just now after activating it. But I can't make sense of it at the moment.
Is it related to the new integrated runtime step or to the current XI sender adapter version or other platform updates...

alex_bundschuh · ‎2025 Apr 08

@ronald_fassl1 it's hard to say what the root cause is, maybe you face an error where the payload can't be retrieved before writing into the dead letter queue, but this can't really happen unless the payload is already empty in the first JMS queue because the payload is saved right at the beginning of the flow, what was the error in the first place? Or the payload comes empty in from the XI inbound processing flow so it might be related to the new XI adapter setting handled by integration flow, there were some issues in the past and a patch has been provided. Are you on the latest version of the XI inbound? It should be 1.0.4. You may update to 1.0.4 and redeploy or revert to 1.0.2 where we do not use the handled by integration flow setting and run some tests, best is to raise an incident ticket, if this is related to the XI adapter then on the support component LOD-HCI-PI-CON-SOAP

Alex

ronald_fassl1 · ‎2025 Apr 08

@alex_bundschuh thank you very much for your valued input!

Error in the first place was the PIP07 scenario specific outbound processing, where a content modifier step failed due to an XPath error (because no payload available).

Pipeline Generic Step01 - Inbound Processing for XI (ok)
Pipeline Generic Step02 - Integrated Messaging Runtime Async (ok)
PIP07 iFlow copied from standard (failed)

There is no error visible in Step01.

Indeed XI inbound was on version 1.0.3. I have updated now to 1.0.4.

I will closely monitor the processing and raise an incident as you suggested if the error persists.

Thanks,

Ronald