Cloud Integration - Configuring Scenario Using the XI Sender Adapter

[mandy_krimmel]

This blog describes how to use the new XI sender adapter, which will be available for customers with the 10-June-2018 release. It describes the configuration options, size limits and monitoring options of your scenario.

Configuring a Scenario Using the XI Sender Adapter

Many customers run existing on-premise backend systems that can connect using the XI 3.0 protocol to an integration broker. In most cases the broker is an SAP NetWeaver PI (Process Integration) or PO (Process Orchestration) system. Now, with the new XI adapter, customers can also connect their on-premise backends to SAP Cloud Integration using the XI 3.0 protocol. Whereas the XI sender adapter can be used to receive messages via the XI 3.0 protocol, the XI receiver adapter can be use used to send out messages using the XI 3.0 protocol.

In this blog, I focus on the XI sender adapter and continue with the sample scenario started for the XI receiver adapter in blog Configuring Scenario Using the XI Receiver Adapter. In this scenario, we will now consume the asynchronous XI message from the backend system. To follow the whole scenario it's easiest for you to start with the sample scenario using the XI receiver adapter and then continue with the XI sender scenario. But you can also consume the scenario for the XI sender adapter independently. In that case, you just configure the confirmation message and send it using transaction SPROXY.

To receive a message via XI 3.0 protocol in Cloud Integration, you need to configure an integration flow with an XI sender adapter and you need to configure the sender backend to send the message to the endpoint the XI sender adapter of the deployed integration flow exposes in Cloud Integration.

Configure the Integration Flow Receiving the XI Message

First, we configure the integration flow in the Web UI, Design section. Create an integration flow, connect the sender participant with the start message event and select the XI adapter.

Configure the XI Sender Channel

First, you choose the Connection tab in the XI sender channel. Configure a path in the Address field. This path will be generated into the endpoint URL, which is called from the sender system to trigger the integration flow execution.



In the Delivery Assurance tab, specify the Quality of Service and, in case of At Least Once and Exactly Once, the Retry configuration settings. Let's check the options in more detail.

To define the Quality of Service, you have the following options: either At Least Once, Exactly Once or Best Effort:

• Best Effort is used for synchronous request-reply scenarios, that means, in case your sender system has a synchronous interface through which you like to send a message to Cloud Integration you use this option. If you select this option, you don't need to configure any additional settings because the message request and the response are processed immediately in Cloud Integration. Therefore, no temporary storage of the message for later retry is involved.


• Exactly Once and At Least Once are both used for asynchronous one-way scenarios, that means if your sender system has an asynchronous interface through which you like to send a message to Cloud Integration you need to use this option. If you configure this option the message is temporarily stored in the Cloud Integration tenant and, if an error occurs, the message is retried from there. The sender gets the successfully received status as soon as the message is persisted in the temporary storage. The difference is that
  
  
  
  • Exactly Once (with version 1.8 or higher of the XI sender adapter in January 2019 release) is ensuring that the XI sender adapter does not process the message if the same XI message ID was already received. It returns a duplicate message back to the sender, but does not process the message in Cloud Integration. This is implemented using an Idempotent Repository storing all the received XI message IDs.
    
    
    
    • Note, that this option can only be used with Data Store as temporary storage because also the Idempotent Repository is based on the database and Cloud Integration cannot support consistent update between JDBC and JMS resources.
    
    
    • Also be aware that the performance is not as good as with the At Least Once option because of the additional check against the Idempotent Repository.
    
    
    • You should use this option only if urgently required, usually the receiver system can detect and handle duplicates. In this case we suggest to use the At Least Once option.
    
    
    
    
  
  
  • Exactly Once (with version up to 1.7 of the XI sender adapter) processes the same XI message multiple times, if the sender sends it again. With this option the receiver should be able to handle duplicates. This option is equivalent to the At Least Once option available in the 1.8 version.
  
  
  • At Least Once (with version 1.8 or higher of the XI sender adapter in January 2019 release) processes the same XI message multiple times, if the sender sends it again. With this option the receiver should be able to handle duplicates. Note, that this option has better performance because no check against the Idempotent Repository has to be done.
  
  
  
  

We configure At Least Once for our sample scenario as we want to receive an asynchronous message. In this case the Temporary Storage needs to be defined. You can select to store the message temporarily either in the Data Store or in JMS Queues:

• When Data Store is selected the message is persisted temporarily in the tenant's database. The stored message can be monitored in the data store monitor.


• If JMS Queue is selected the message is stored in a JMS queue on the connected JMS broker. Note, that the JMS Queue option is the recommended option because it is more performant but is only available if you have an Enterprise edition or Integration Suite for the tenant. (You can find more details in the in blog ‘How to activate Message Broker’.)

If Data Store is configured as Temporary Storage, the following additional settings need to be defined:

• Retry Interval: Here you define after which time a retry shall be triggered if there was an error during message processing.


• Exponential Backoff: If the flag is selected, which I would recommend, the retry interval is doubled after each unsuccessful retry.


• Maximum Retry Interval: Here you enter the limit to avoid endless increase of the retry interval caused by the exponential backoff setting. This field is only available is Exponential Backoff is selected. The recommendation is to either keep the 60 minutes or even increase it if this is acceptable by the scenario.


• Encrypt Message During Persistence: If this option is selected the message is encrypted in the data store during temporary storage. Note that this is recommended if the message contains sensitive data. However, the drawback is that this setting reduces the performance slightly.


• Note that no configuration option is available for dead-letter queue handling. If the data store is used as temporary storage the out-of-memory handling is always active. Which means that after the 3rd retry caused by an out-of-memory a message is blocked and taken out of further processing.

If JMS Queue is configured as Temporary Storage the following additional settings need to be defined:

• Retry Interval: Here you define after which time a retry shall be triggered if there was an error during message processing.


• Exponential Backoff: If the flag is selected, which I would recommend, the retry interval is doubled after each unsuccessful retry.


• Maximum Retry Interval: Here you enter the limit to avoid endless increase of the retry interval caused by the exponential backoff setting. This field is only available is Exponential Backoff is selected. The recommendation is to either keep the 60 minutes or even increase it if this is acceptable by the scenario.


• Dead-Letter Queue: You should select this option to handle potential out-of-memory situations caused by large messages. If selected,  a message is blocked and taken out of further processing after the 3rd retry caused by an out-of-memory. More details about the dead letter queue handling can be found in blog How to configure Dead Letter Handling in JMS Adapter.


• Encrypt Message During Persistence: You define if the message is to be encrypted in the data store during temporary storage. Note that this is recommended for sensitiv data but it reduces the performance slightly.

Starting with version 1.6 of the XI sender adapter, which is available with the 28-October-2018 version, you have the option to define size restrictions for inbound processing. In the Conditions tab adjust the default values for the Body Size, if required:

Configure Transaction Handling

You need to configure the correct transaction manager in the integration process for transactional end-to-end processing. Our process does only contain an XI Sender and no data store flow steps or JMS adapters; so we don’t need a specific transaction handler. Select the integration process and switch to the Processing tab. In Transaction Handling drop down select Not Required.



More details about the different transaction handling options and existing limitations are described in the blog ‘How to Configure Transaction Handling in Integration Flow’.

Configure the Outbound Channel

Configure the integration flow with the outbound channel required by your scenario. For the sample scenario we use the sftp adapter and send the message to a directory on the SFTP server. But you can use any other adapter as well.

Deploy the Integration Flow

Now you can deploy the integration flow. Afterwards, check if the integration flow was started successfully in the Manage Integration Content monitor.

Get the Endpoint URL to be called

To be able to call the integration flow from the sender system, you need to know the endpoint URL to be used. For this check the Endpoints tab in the Manage Integration Content monitor. There the URL is shown and can be copied to the clipboard using the Copy option.

Configure the Sender System to Send the XI Message to Cloud Integration

The next step is to configure the sender backend to send the message to the Cloud Integration tenant. You need to configure the Integration Engine in the backend using transaction SXMB_ADM -> Integration Engine Configuration.

Configure Local Integration Engine 

If you did not already configure the local Integration Engine executing the XI Receiver blog you need to configure the Local Integration Engine in the backend using transaction SXMB_ADM -> Integration Engine Configuration. Here the Role of the Business System defined should be Application System.

Configure SLD

To send XI messages you need to make sure the SLD is configured correctly. Execute transaction SLDCHECK in the application backend to verify this.

Define RFC Destination with Integration Server URL

There are different options available to define the integration server URL to send messages to via XI 3.0 protocol, either define it for the whole backend system or interface-specific. You can find detailed information on this in the online help for the application system you are using.

To not interfere with any other scenarios in the backend, we will use the interface-specific configuration and define the XI endpoint URL to be called only for a specific interface.

Use transaction SM59 to create a new HTTP Connection to an external server. Define the target host and path prefix according to the endpoint URL you copied from the Endpoints tab. The path prefix is composed according to the following pattern: /cxf/<address field in XI sender adapter>.



In the Logon & Security tab, define SSL and the SSL Client Certificate to be used for the connection. If Basic Authentication is to be used, enter User and Password.

Test the connection using the Connection Test button. You should get an HTTP 500 with the response that an internal error occurred with reference to an MPL. This is because we did not send a payload along with the request. You should not get any SSL error or HTTP 40* errors, in this case you need to check the certificate setup and potentially the user-certificate mapping in the Cloud Integration tenant or the assignment of the role ESBMessaging.send in Cloud Platform role management for the login user. More details about setting up the inbound authentication can be found in the blog 'How to Setup Secure HTTP Inbound Connection with Client Certificates'.

Define Sender/Receiver Definition

Use transaction SXMSIF to define a sender/receiver definition for the interface you want to use. Configure * for Service to make sure the configuration is valid for all sender and receiver services.



In the sample scenario we use the interface FlightBookingOrderConfirmation_Out from namespace http://sap.com/xi/XI/Demo/Airline from the PI Demo Examples as we want to send the confirmation after we got the flight booking configured in the scenario using the XI receiver adapter. But as already mentioned, you can also use any other interface available in your backend to try this out.

Activate Interface-Specific Endpoint

Use transaction SXMB_ADM -> Integration Engine Configuration --> Specific Configuration to define a new parameter with the following settings:

• Category: RUNTIME


• Parameters: IS_URL


• Subparameters: Select the Sender/Receiver ID you defined in the last step.


• Current Value: dest://<name of the RFC destination you defined in SM59>

With this, the system will send XI proxy calls for the interface FlightBookingOrderConfirmation_Out to the Cloud Integration tenant.

Test Sending a Message to the Cloud Integration Endpoint

Now we can test the configuration. The easiest way to do this is using transaction SPROXY. In SPROXY select the Service Interface FlightBookingOrderConfirmation_Out in namespace http://sap.com/xi/XI/Demo/Airline. In the menu, choose Proxy -> Test. In the Test Service Consumer dialog leave the settings as they are and select Execute:



A screen with a sample test message is opened, press Execute and afterwards select Extras -> Trigger Commit Work. The request is sent to the Cloud Integration tenant. You can now monitor the processing as described in the next section.

Check in transaction SXMB_MONI that the message was sent successfully. Furthermore, check in Cloud Integration monitoring that the two message was processed. More details about the monitoring in Cloud Integration can be found below in the Monitoring section.

Test the Whole E2E Scenario

If you started with the flight booking scenario using the XI receiver adapter, you can now test the whole end-to-end scenario. Trigger the scenario by putting a file into the sftp folder and check the monitoring in your Cloud Integration tenant and in transaction SXMB_MONI in the backend.

In Cloud Integration monitoring you should find two message processing logs for the XI receiver integration flow sending the asynchronous request to the backend and one message processing log for the XI sender integration flow receiving the asynchronous response message:



In transaction SXMB_MONI there should be two XI messages; one for the request received and one for the response sent:

Monitoring

To check the processing in Cloud Integration detail, you can use the monitoring tools provided by Cloud Integration.

Message Monitoring

Message processing can be checked in detail in the section Monitor Message Processing. Select the message you sent and check the processing status. There is one message processing log for the whole processing, containing the inbound messaging that saves the message to the data store/JMS queue and the outbound processing from the data store/JMS queue and sending it to the receiver system.

In the event of a failure the message will go into status Retry. The message will then be retried as configured in the XI sender channel. The detailed error information is available under the link Message Processing Log.

Monitor XI Message ID in Idempotent Repository (For Scenarios using the Exactly Once Option)

For the monitoring of the already received message ID, which are persisted in the Idempotent Repository, an OData API is available. You may use it to find out if a specific XI message was already processed.

For example, you can use the Postman tool to call the OData API for the Idempotent Repository component.

Use the following URL to retrieve the source ID for an XI message ID:

https://<tmn-server>/api/v1/IdempotentRepositoryEntries?id=<xi-message ID>

example:

https://server/api/v1/IdempotentRepositoryEntries?id=00505615-0124-1ee8-b8ea-9aace1a9f929

The call returns 202 if the XI message ID was already processed or 404 if it is not contained in the Idempotent Repository. In case the entry exists in the Idempotent Repository, the Creation and Expiration Times are provided.

Data Store Monitor (For Messages Stored in Data Store)

You can find the messages that are stored temporarily during processing in the Data Stores tile in the Manage Stores section in the monitoring dashboard. During successful processing, the messages are removed immediately after the processing was completed. In case of an error, the messages are kept in the data store to trigger the retry from there.



The used data store name is auto-generated according to the following pattern: <participant name>_< XI channel name>. Below the data store name the <integration flow name>/XI is shown. In the monitor all messages are listed with a status. Two statuses exist:

• Status Waiting means that the message waits for consumption.


• Status Overdue indicates that the message was not consumed and processed in the due time, which is 2 days in case of the XI adapter.

In the monitor the Message ID is shown as a link. If you select the link the respective message processing log is opened in the message monitor.

Note that data stores that do not have messages persisted at the time the monitor is opened are not shown in the monitor. This is because data stores are created on the fly in message processing if required and removed after the last message was removed from data store.

Lock Monitor (For Messages Stored in Data Store)

In case the message was involved in several node outages, for example caused by an out-of-memory, the message gets locked in the Lock Monitor. Those messages are not retried any longer. Notice that you will probably not see any entry in this monitor during testing your sample scenario, as this entry will only be permanently available in case of a runtime node outage. So, see this section more as additional information.

The entry will be generated with the following parameters:

• Component: XI


• Source: XI_<integration flow name>.<participant name>_<channel name>


• Entry ID: corresponds to the ID shown in data store monitor

You need to check if the message was the root cause if the error, maybe it is just too large to be processed. Then you need to ask the sender to send the data in smaller messages. Afterwards you can delete the message in the Data Store monitor and then also delete the lock in the Manage Locks monitor. If you are confident that the message did not cause the outage you can release the lock in the Manage Locks monitor by using the Release button.

Queue Monitor (For Messages Stored in JMS Queue)

You can find the messages that are stored temporarily during processing in the Message Queues tile in the Manage Stores section in the monitoring dashboard. During successful processing the messages are removed immediately after the processing was completed. In case of an error the messages are kept in the JMS queue to trigger the retry from there.



The used JMS queue name is auto-generated according to the following pattern: XI.<Integration Flow Name>.<Channel Name>.<Guid>. In the monitor all messages are listed with a status. Four statuses exist:

• Status Waiting means that the message waits for consumption.


• Status Failed indicates that the last processing or retry of the message ended with an error.


• Status Blocked comes into play if dead-letter queue is configured in XI channel and if the message was involved in several node outages and so is taken out of processing. See dead-letter queue handling above.


• Status Overdue indicates that the message was not consumed and processed in the due time, which is 2 days in case of the XI adapter.

You can see the number of retries that have already been executed and the time of the next retry. You can also trigger an immediate retry of the message or delete specific messages if you no longer want them to be processed.

Queue Deletion

If you do no longer need a specific queue(for example if you un-deploy the integration flow), you have to delete the queue manually in the Queue Monitor if the queue contains messages. The queue cannot be deleted automatically because this could cause data loss. Only the scenario owner knows if the messages can

To find unused queues in the monitor you may use the checks described in the blog 'Checks in JMS Message Queue Monitor'.

JMS Resource and Size Limits

If you are using JMS for temporary storage you need to keep in mind that the connected JMS messaging instance, that is used has limited resources. The Enterprise or Messaging License sets a limit on the queues, storage and connections that you can use in the messaging instance. The limits are described in more detail in the blog 'JMS Resource and Size Limits'.

Important Considerations and Limitations

When using the XI adapter, there are some very important facts you need to be aware of:

Use JMS option if JMS is available in the tenant

To avoid additional load on the database and to have better performance you are strongly recommended to use the JMS option for temporary storage.

Using the Headers set by the XI Sender Adapter Requires to Allow the Required Headers

If you want to use the headers set by the XI Sender adapter in the integration flow configuration you need to add the required headers in the Allowed Header(s) field in the Runtime Configuration of the integration flow. For details check the blog Configuring Scenario with XI Sender handling Multiple Interfaces.

Changing Storage Between Data Store and JMS May Cause Data Loss:

As the adapter stores messages for retry in case of an error either in a JMS queue or in the Data Store, you need to keep in mind that changing this setting for an already running scenario may cause loss of data. This is because there can still be messages in the old storage, which will not be retried anymore after the storage changed. Those messages would stay in the data store or JMS queue until they are deleted (see field Retain Until in Data Store or Queue monitor).

Because of this, be very careful with such changes in productive scenarios. Make sure there are no message left in the 'old' storage before changing this setting.

Changing Participant or Channel Name May Cause Data Loss:

As the data store or JMS queue name for retry is generated based on the participant and channel name, a new data store or JMS queue would be generated if you change one of those names. As there can still be messages in the old storage, these messages will not be retried anymore after the storage changed. Those messages would stay in the data store or JMS queue until they are deleted (see field Retain Until in Data Store or Queue monitor).

Because of this be very careful with such changes in productive scenarios. Make sure there are no message left in the 'old' storage before changing participant or channel name. With the 30-September-2018 update you have the option to move all messages from one JMS queue to another, if required. This option would help you in case you urgently need to change the channel or participant name or if you have done it without knowing the constraint that a new queue would be generated.

To move messages from one JMS queue to another use the Message Queues monitor. In the single line actions for the ‘old’ queue select the Move action:



In the Move Messages dialog select the newly generated queue and choose Move. All messages of the old queue are now moved to the new one.



Note that there is currently no such move option for the data store. The blog will be updated as soon as this is available.

Wrong transaction handling configuration may cause inconsistent data and/or data loss:

The two storage options, JMS Queue and Data Store, require different transaction handlers. Because of this it is crucial that you define the correct transaction handler:

• JMS Queue: If a JMS Queue is used for temporary storage in the XI receiver adapter and the XI adapter is used in a sequential multicast or splitter pattern you need to configure the Transaction Handling as Required for JMS.


• Data Store: If the Data Store is used for temporary storage in the XI receiver adapter and the XI adapter is used in a sequential multicast or splitter pattern you need to configure the Transaction Handling as Required for JDBC.


• For the XI sender adapter no transaction handler is required.


• For the XI receiver adapter no transaction handler is required if the XI adapter is not used in a sequential multicast or in a split scenario.


• As there is no distributed transaction support in Cloud Integration you cannot combine JMS and JDBC transactions. This means that you cannot ensure transactional behavior in scenarios using the XI receiver adapter with JMS storage in multicast scenarios together with flow steps that need a JDBC transaction handler, like for example Data Store or Write Variables.

More details about the transaction handling configuration and the constraints in configuration can be found in blog ‘How to configure Transaction Handling in Integration Flow‘.

The XI adapter currently has some limitations:

• No explicit retry handling via Retry Header in Exception Sub-Process possible possible yet. -> Solved, see blog Configuring Explicit Retry in Exception Sub-Process for XI Adapter Scenarios.


• MPL status stays in Retry even if the message is deleted from JMS queue or data store. Nevertheless in runtime no retry is executed anymore. -> solved, messages are now getting status Cancelled when the corresponding message is manually deleted in JMS or Data Store monitor. Note that the deletion of a message after the expiry date does not lead to Cancelled messages because the delete is directly done on the database/JMS broker.


• XI receiver adapter is currently neither allowed in Request-Reply nor in Send step. -> Solved, see blog Usage of XI Adapter in Send and Request-reply Step.


• Acknowledgements are not supported


• Attachments are not supported -> solved with the latest versions of the XI sender and receiver adapter (February 2021)


• Packaging is not supported -> solved with the latest versions of the XI sender adapter (April 2021)


• Fault Messages are not supported


• ExactlyOnceInOrder (EOIO) is not supported.

The blog will be updated regularly with the new features.