Enhancing SAP CPI with Custom Payload Logging using S4HANA On Prem and a UI5 Dashboard

[KarthikBangeraM]

In the dynamic world of SAP Cloud Integration (CPI), efficient logging mechanisms are crucial for maintaining robust and reliable integration processes. While CPI offers various built-in options for message monitoring and logging, these may not always meet specific project requirements. In this blog, we will explore a custom logging approach that empowers developers to capture and manage payloads more effectively. There are various logging options already available by using Groovy scripts, Data Stores, and external logging services like Splunk.

However, we will try to create a tailored logging solution that enhances visibility and troubleshooting capabilities within your CPI environment, utilizing the existing SAP OnPrem solution in your landscape. Join me as we get into the details of this custom strategy, designed to optimize your integration workflows.

Prerequisites-

1. CPI Subscription – NEO or CF
2. S4HANA On Prem
3. WebIDE or BAS

Limitations-

1. Can only log one payload per Iflow as per the current design. This can be further extended to support multiple step payloads in the upcoming versions.
2. Only Supports logging and display of XML, JSON and CSV payloads for now.

The below solution diagram gives you a holistic view of the applications involved in this design and their roles in each of the steps.

Solution Diagram-

Overall Steps-

1. CPI Logger Iflow to call an Inbound Proxy – This Iflow can be called by any Iflow in your CPI tenant which needs to log its payload.
2. Inbound Proxy to save payload to AL11 – This Inbound proxy will be called by the CPI Logger Iflow to save the payload to an AL11 directory in S4HANA On Prem
3. CPI Dashboard to consume CPI’s Monitoring API and to display the payload – The UI5 app connects to CPI monitoring service to read and display a monitoring overview screen (much like the traditional SAP PO). It then allows you to open any individual message and display the corresponding payload of the message. This uses a standard set of APIs which allows the UI5 dashboard App to fetch the CPI monitoring data
4. OData Service to enable our CPI Dashboard app to read the payload from AL11 directory – This OData service will enable you to read any given AL11 payload using its Message GUID as an input.

This is how the end solution will look like-

Open the CPI Dashboard UI5 app and search for an overview of messages within a given time period and click on the resultant row with Iflow (ZPayloadLogger_Src) whose payloads you want to view.

XML Payload-

This will open a list of all the messages for that Iflow within that given period. Click on the first row.

This will take you to a new page which will display the Iflow Name and the Message GUID. The payload will then be displayed in a text area below. You also have an option to download the payload using the “Download Payload” button. This is an example of an XML payload.

Below is how the JSON and CSV payloads will be displayed-

JSON-

CSV-

If a certain CPI Iflow isn’t using this functionality to log its payload-

GIT Repository of the CPI Dashboard App-

Git Repo for Dashboard App - NEO 

Please note that the above project has been built in WebIDE for NEO. I have created a Repository for Cloud Foundry as well. You can modify the xs-app.js and mta.yaml files with your respective destination references-

Git Repo for Dashboard App - CF 

You are welcome to suggest improvements and new features in the App as you explore it.

We will now go through all the building blocks of this application-

Steps 1 and 2 – CPI Iflow calls an SAP Inbound Proxy and this proxy program saves the payload in the AL11 directory

Inbound Proxy Program to be called by CPI-

The inbound proxy program will be consumed by CPI to send payloads to be logged from CPI into SAP. The program saves the received payloads in a predefined AL11 directory with <MessageGUID>.<Extension> format as shown in the below screenshot.

Inbound Proxy Program Details-

Service Provider-

Implementing class-

Implementing method-

Inbound Proxy Code-

The below Inbound proxy code accepts Message GUID and Payload type as inputs from the consumer (ZPayloadLogger Iflow in our case) and calls the Function module  Z_SAVE_PAYLOAD_AL11 which saves the payload to the predefined AL11 directory path.

method zii_si_cpipayload_logger_in~cpipayload_logger.
*** **** INSERT IMPLEMENTATION HERE **** ***

    data: lv_result type string,
          lv_payloadcontent type string,
          lv_filename type string.

    data: lv_start_index type i,
        lv_end_index type i,
        lv_cdata_length type i,
        lv_cdata_tag type string.

    data: msg_guid     type c length 50,
          payload_type type c length 30.


loop at input-mt_cpipayload-payload into data(lv_payload).

  lv_filename = lv_payload-message_guid && '.' && lv_payload-payload_type.

  call function 'Z_SAVE_PAYLOAD_AL11'
  exporting
    lv_payload_data = lv_payload-payload_content
    lv_filename = lv_filename
  importing
    lv_status = lv_result.

  msg_guid = lv_payload-message_guid.
  payload_type = lv_payload-payload_type.

     endloop.

  endmethod.

Function Module – Z_SAVE_PAYLOAD_AL11

FUNCTION Z_SAVE_PAYLOAD_AL11.
*”----------------------------------------------------------------------
*”*”Local Interface:
*”  IMPORTING
*”     REFERENCE(LV_PAYLOAD_DATA) TYPE  STRING
*”     REFERENCE(LV_FILENAME) TYPE  STRING
*”  EXPORTING
*”     REFERENCE(LV_STATUS) TYPE  STRING
*”----------------------------------------------------------------------

DATA: lv_file_path TYPE string,
            lv_directory TYPE string.

  “ Set the directory path where the file will be saved
  lv_directory = ‘<Your AL11 Filepath>’.

“ Concatenate the directory path and file name
  CONCATENATE lv_directory lv_FILENAME INTO lv_file_path.

  OPEN DATASET lv_file_path FOR OUTPUT IN TEXT MODE ENCODING DEFAULT.

 TRANSFER lv_payload_data TO lv_file_path.

 CLOSE DATASET lv_file_path.

  “ Check if the file was successfully saved
  IF sy-subrc = 0.
    lv_STATUS = ‘Payload saved to AL11 directory:’.
  ELSE.
    lv_STATUS = ‘Failed to save payload to AL11 directory.’.
  ENDIF.

ENDFUNCTION.

Expected Payload by Inbound Proxy- (Data under xml element “PayloadContent” can either be xml, json or csv) –

Below is an example of an xml payload that can be sent into the Inbound proxy from CPI’s Iflow.

<n0:MT_CPIPAYLOAD xmlns:n0='http://sap.com/Basis/CustomFields/Generated/*'>

  <Payload>

    <PayloadContent><![CDATA[    <test>

      <Field1>Value2</Field1>

      <Field2>Value3</Field2>

      <Field3>Value4</Field3>

    </test>]]></PayloadContent>

    <MessageGUID>521100b2-66e4-4f39-b4c5-451cfaab057d</MessageGUID>

    <PayloadType>XML</PayloadType>

  </Payload>

</n0:MT_CPIPAYLOAD>

CPI Flows

We have 2 CPI flows here,

1. ZPayloadLogger_Src – This represents any source flow which wants to log its payload by referring its Message GUID.
2. ZPayloadLogger – This represents the actual CPI logger flow which can be called by any Iflow in your landscape (like ZPayloadLogger_Src flow) that wants to log its payload.

Expected Input Payload into ZPayloadLogger Flow:

Consumers of this Iflow are required to simply send the payload in XML, JSON or CSV format within the request body and pass the below 2 headers-

       1. MessageGUID

       2. PayloadType

ZPayloadLogger_Src IFlow

As shown above, the original Iflow (ZPayloadLogger_Src) which wants to log its payload, needs to capture it’s MessageID using camel header “SAP_MessageProcessingLogID” and then send it in the message header to the CPI Logger Flow (ZPayloadLogger). It also needs to send a header called “PayloadType”, which can be xml, json or csv, depending on the type of the body sent to ZPayloadLogger IFlow.

ZPayloadLogger IFlow-

XI Channel connecting to S4HANA OnPrem via Cloud Connector-

Below are the key channel configurations-

Refer below blog for more details on how to connect CPI to SAP OnPrem via Proxy method-

Cloud Integration - Configuring scenario using the XI receiver adapter 

Script to construct the expected Inbound Proxy Payload-

The below Groovy script within the ZPayloadLogger Iflow constructs the payload required by the Inbound Proxy program.

import com.sap.gateway.ip.core.customdev.util.Message
import groovy.xml.MarkupBuilder

def Message processData(Message message) {
    //Get headers
    def headers = message.getHeaders()
    def messageGuid = headers.get("MessageGUID")
    def payloadType = headers.get("PayloadType")

    //Get payload content from message body
    def payloadContent = message.getBody(String)

    //Create XML structure
    def writer = new StringWriter()
    def xml = new MarkupBuilder(writer)
    xml.'n0:MT_CPIPAYLOAD'('xmlns:n0': 'http://sap.com/Basis/CustomFields/Generated/*') {
        Payload {
            PayloadContent {
                mkp.yieldUnescaped "<![CDATA[$payloadContent]]>"
            }
            MessageGUID(messageGuid)
            PayloadType(payloadType)
        }
    }

    // Set the output message body
    message.setBody(writer.toString())

    return message
}

Step 3 – Consuming CPI’s monitoring API from CPI Dashboard App

Below is the CPI monitoring API endpoint to fetch all the monitoring logs into the UI5 app.

“/v1/MessageProcessingLogs”

Below is the link to the Business Accelerator Hub which gives an in-depth API reference and guidelines to consume it.

Reference - https://api.sap.com/api/MessageProcessingLogs/resource/Logs

Destination service in BTP-

Below is the destination that needs to be created in BTP to connect to the CPI service.

Then, refer this destination within your neo-app.json (NEO) or xs-app.json (CloudFoundry) file of the UI5 app.

Step 4 – OData service to be called by the UI5 Dashboard App to get the saved payloads using MessageGUID

The payloads stored in AL11 directory need to be accessed by our CPI Dashboard app. For this reason, we will create a Function module which takes MessageGUID as an input and returns the resultant payload from the AL11 directory.

Function Module Code-

The below code accepts Message GUID as an input and responds back with the Payload and Filetype.

FUNCTION Z_GET_PAYLOAD.
*"----------------------------------------------------------------------
*"*"Local Interface:
*"  IMPORTING
*"     VALUE(LV_GUID) TYPE  STRING
*"  EXPORTING
*"     VALUE(LV_PAYLOAD) TYPE  STRING
*"     VALUE(LV_FILETYPE) TYPE  STRING
*"----------------------------------------------------------------------

DATA:
  lv_al11filepath TYPE string,
  lv_line TYPE string,
  lv_dir TYPE eps2filnam,
  p_file_n TYPE localfile,
  lv_extension TYPE string,
  lv_valid_extension TYPE abap_bool,
  lv_extensions TYPE TABLE OF string.

lv_dir = '<AL11 Directory Path>'.

lv_PAYLOAD = ''.

APPEND 'csv' TO lv_extensions.
APPEND 'json' TO lv_extensions.
APPEND 'xml' TO lv_extensions.

LOOP AT lv_extensions INTO lv_extension.
  lv_al11filepath = lv_dir && lv_guid && '.' && lv_extension.

  p_file_n = lv_al11filepath.

  OPEN DATASET lv_al11filepath FOR INPUT IN TEXT MODE ENCODING DEFAULT.
  IF sy-subrc = 0.
    lv_filetype = lv_extension.
    lv_valid_extension = abap_true.

    DO.
      READ DATASET p_file_n INTO lv_line.
      IF sy-subrc <> 0.
        EXIT.
      ENDIF.
      lv_PAYLOAD = lv_PAYLOAD && lv_line && cl_abap_char_utilities=>cr_lf.
    ENDDO.

    CLOSE DATASET p_file_n.
    EXIT.
  ENDIF.
ENDLOOP.

IF lv_valid_extension <> abap_true.
  lv_PAYLOAD = 'No valid file found'.
ENDIF.

IF sy-subrc <> 0.
  lv_PAYLOAD = 'Error occurred while reading file'.
ENDIF.

ENDFUNCTION.

Next step is to expose this Function module as an OData service, which will then be consumed by our Dashboard App. You can follow the below blog for your reference for this.

Steps to build an RFC based OData service with multiple selection values 

This is a screenshot of how your generated OData service will look-

Testing the OData service giving Message GUID as a reference-

The xml payload is returned in response to the OData request call.

Below is the destination service configuration in BTP to connect to this OData service-

Then, reference the above destination within your neo-app.json (NEO) or xs-app.json (CF) file as shown below-

As shown above, once all the steps shown in the Solution Diagram are implemented, you should be able to run your Dashboard app and view payloads of IFlows that are using this functionality.

Demo-

Now that we have completed all the required configurations, we will go ahead and demonstrate the functionality using a message triggered from Postman client.

Send a sample xml payload using Postman client to the endpoint of ZPayloadLogger_Src Iflow in CPI-

Message processed as per CPI message monitor- Note that ZPayloadLogger_Src is the actual flow whose payload we want to log, whereas ZPayloadLogger flow is the one which calls our Inbound Proxy program to store the payload within SAP’s directory.

Inbound proxy message processed successfully in SAP-

Payload content in SAP- Shows the actual xml file within CDATA tag.

File saved in AL11 directory-

XML content displays as expected in AL11 directory-

CPI Dashboard App – Displays 2 messages in overview page. Click on ZPayloadLogger_Src.

This opens the message monitor window. Notice that the Message GUID matches the one from our CPI and SAP monitoring screens. Click on the message.

This opens the message details page which finally displays our XML message.

Message displayed on successfully downloading the message.

Click on the downloaded payload-

Payload displayed in our xml editor-

We now arrive at the end of this blog post. This solution effectively addresses the need for efficient payload extraction in CPI, offering a practical and user-friendly tool for integration monitoring. The integration of CPI, ABAP, and UI5 showcases a comprehensive approach, leveraging the strengths of each technology to create a cohesive solution.

Comments and suggestions for improvement are welcome.