Cloud Integration Pipeline - Datastore Extension UI

[lisarizzo]

⚠️  Disclaimer: The UI described in this blog post is not an official product by SAP. If you are interested in using the UI, you can find the published code and set up instructions on the SAP-samples GitHub here: https://github.com/SAP-samples/cloud-integration-pipeline-datastore-extension-ui

Purpose and background

The pipeline in Cloud Integration is a well-established framework for asynchronous integrations. A blog post to explain the starting points for the pipeline can be found here.

In addition to the pipeline, we offer the datastore extension, which simplifies error handling via automatic retry jobs and APIs for resending and cancelling messages. For details on the datastore extension, please refer to this blog post. The datastore extension package provides automatic restart capabilities with Restart Jobs that are based on pre-delivered IFlows. In addition, it provides on-demand operations that require the use of API client tools such as Postman or Bruno. This is technical and error-prone to handle from a user perspective.

The UI introduced in this blog post is designed to provide a simplified and user-friendly approach to access these on-demand APIs.

On a technical level, it is relevant to note that the UI does not call Cloud Integration APIs directly. It calls IFlows of the Datastore Extension package, which are on-demand API Operations (Manage Retries and Manage Datastore Entries). It triggers a corresponding IFlow and operation and then displays the response.

This blog article introduces and explains the functionality of the Cloud Integration Pipeline Datastore Extension UI.

Overview of key features

The current version of the UI provides these capabilities, which are shown again in more detail in the “How to use the UI” section.

• Visual overview of all available datastores relevant for the pipeline reprocessing, their messages and overdue messages
• Search and filter capabilities to quickly locate specific datastores
• Visual overview of all messages in each datastore
• Search and filter capabilities to quickly locate specific messages
• Action to restart messages from the UI
• Action to move messages to another datastore or a no-retry datastore when located in a retry datastore
• Action to delete/cancel messages 
• Detailed view of message content (error, headers, payload)
• Possibility to perform batch actions to restart, move, cancel several messages at once

How to use the UI

Datastore overview

The first screen of the UI shows a table with all the datastores, the number of messages, and the number of overdue messages. You can also search and filter the table for specific information. 

Datastore detail view and overview of messages

The second screen shows details of the selected datastore and contains a table with all the messages. 

One or more messages can be selected in the table and moved to another datastore or to a no-retry datastore. The messages can also be restarted or deleted from here.  

There are sorting capabilities available for the datastore entries table. They can be found behind the gear icon ⚙️. For example, the table can be sorted by the number of restarts and the timestamp of error, ascending or descending. 

There is also a filter hidden behind the same gear icon ⚙️.

Message detail view

When clicking on one of the rows in the datastore entries table the user is directed to a third screen with details about the error, headers and payload. If, after inspecting, the user decided what to do with the message it is also possible to perform the same move, delete and restart actions from this screen with the buttons in the top right corner.

Outlook & how to implement the UI

This project is available as an open-source project on the SAP-samples GitHub organization: https://github.com/SAP-samples/cloud-integration-pipeline-datastore-extension-ui . Here you can find the repository and a detailed implementation guide. If you notice any issues or have further ideas feel free to contribute to the project. You can also reach out to @lisarizzo or @mike_sibler  if you have any further questions.

[MateuszPiotrowski]

@lisarizzo 
It’s fantastic to see the Pipeline Concept expanding so rapidly! The new DataStore UI and previous Partner Directory UI really highlight how feature-rich the solution has become.

BR, Mateusz

[Steve-Kangethe]

This is a great addition and it works quite well without having to do it all manually in Postman.  Well done!

[philippeaddor]

I love the UI! It's well designed. Just the filtering function behind the gear button is a bit cumbersome to access and it's either not clear to me how it works or it doesn't work as expected...

By the way, I run the app locally on my Windows machine using Docker Desktop with a batch file to start one Container per CI environment. I start the containers on three different ports (using Docker's port forwarding) to distinguish our three CI tenants (since the tools doesn't support multiple tenants yet). That works well and saves us the effort of deploying/running it on a VM or on BTP Workzone (and pay for those) and resolves the problem with the lacking user authentication as well. 

Wish it would become officially part of CI in the future, maybe with a setting to switch it on or off, depending if a customer uses the Pipeline Concept or not. 

[HarshPatelV]

Excellent enhancement! Providing a dedicated UI for the Cloud Integration Pipeline Datastore Extension makes message reprocessing much more accessible for operations teams. The ability to search, filter, perform batch actions, and view detailed message content directly from the UI significantly simplifies retry and error-handling workflows without relying on external API tools like Postman or Bruno. Thanks for sharing this practical improvement and the detailed walkthrough.