Introduction
Organisations frequently need to synchronise large volumes of business data, such as master data, transactional records, financial information, and order data, between enterprise systems. Such integrations must be designed to process high data volumes efficiently while ensuring reliability, recover-ability, and minimal impact on source and target system performance.
This article describes an orchestration-based synchronisation framework built using SAP Integration Suite. The solution leverages JMS queues, pipeline integration flows, and parallel processing principles to transfer large datasets reliably and efficiently between source and target systems.
In this use case, the source system contains master data records that need to be synchronised with a target system without impacting operational performance.
The source system exposes APIs that provide the following capabilities:
- Retrieve the total number of records available for synchronisation.
- Retrieve a specific set of records based on page number and page size for a given date range.
By utilising SAP Integration Suite as the orchestration layer, large datasets can be split into manageable batches and processed independently, enabling scalability, fault tolerance, and efficient resource utilisation.
High-Level Solution Architecture
Process Flow Overview
- Initiation: Scheduled Trigger
The synchronisation process begins with a scheduler that triggers the workflow at predefined intervals based on business requirements.
Once triggered, a message is placed into an initial JMS queue. This design provides decoupling between process stages and ensures message persistence. In the event of temporary connectivity failures or unexpected processing issues, the message remains available for automatic retry and reprocessing.
Key Benefits
- Scheduled execution
- Reliable message persistence
- Automatic recovery from transient failures
- Decoupled processing architecture
- Planning: Determine Processing Scope and Create Jobs
After the workflow is initiated, the planning phase identifies the volume of data that must be synchronised.
During this step:
- The source system API is invoked to retrieve the total record count.
- The total number of pages is calculated based on the configured page size.
- Required processing parameters are generated, including page number and page size.
- A separate job message is created for each page of records.
- All generated jobs are placed into a JMS queue for execution.
This approach converts a large synchronisation activity into multiple smaller and independently executable jobs.
Example
If:
- Total Records = 1,000,000
- Page Size = 10,000
Then:
- Total Jobs = 100
Each job is responsible for processing one page of records.
Key Benefits
- Workload distribution
- Improved scalability
- Independent processing of batches
- Better system resource management
- Execute Jobs from the Queue
Once jobs are available in the JMS queue, the execution flow starts processing them immediately.
For each job:
- The corresponding message is consumed from the JMS queue.
- The source API is called using the assigned page number and page size.
- The requested batch of records is retrieved.
Using JMS as the processing backbone provides resilience against temporary API failures. If connectivity issues occur during execution, the message remains available for retry according to the configured retry mechanism.
Key Benefits
- Reliable batch processing
- Automatic retries
- Failure isolation
- Parallel execution capability
- Retrieve Records and Deliver to the Target System
After successfully retrieving a batch of records from the source system, the data is forwarded to the target system through the SAP Integration Suite pipeline framework.
The processing utilises:
- Pipeline Generic Step 06 – Outbound Processing
- Pipeline Generic Step 08 – XI Outbound Processing
Step 06 provides built-in retry capabilities and reliability handling. If message delivery fails during Step 08 processing, the framework automatically retries transmission based on the configured retry settings.
This layered design ensures guaranteed delivery while maintaining high throughput for large volumes of data.
Key Benefits
- Reliable target system delivery
- Automatic retry handling
- Reduced data loss risk
- Standardised outbound processing framework
Solution Benefits
The proposed orchestration pattern provides several advantages for enterprise-scale integrations:
Scalability
Large datasets are divided into smaller processing units, enabling efficient handling of millions of records.
Performance Optimisation
Paginated data retrieval minimises load on the source system while reducing memory consumption within the integration platform.
Reliability
JMS-based processing ensures message persistence and recovery from temporary failures.
Parallel Processing
Multiple jobs can be processed simultaneously, significantly reducing overall synchronisation time.
Re-usability
The framework can be reused for synchronising:
- Master Data
- Order Data
- Financial Data
- Customer Data
- Product Data
- Transactional Records
Monitoring and Support
Each job represents an individual processing unit, simplifying operational monitoring, troubleshooting, and error recovery.
Conclusion
By combining SAP Integration Suite Pipeline Flows, JMS queues, and pagination-based processing, organisations can build a scale able and resilient synchronisation framework capable of handling high-volume data transfers efficiently. The solution minimises dependency on source system performance, provides built-in recovery mechanisms, supports parallel processing, and ensures reliable delivery to downstream applications.
This orchestration pattern serves as a reusable enterprise integration blueprint for synchronising large-scale master data, transactional data, financial records, and other business-critical information across heterogeneous systems.
Bluesky
