Performance bottleneck in Subscriber Logging Table

[ACARNEVALE]

We are experiencing poor performance while extracting data from CARAB source system towards SAP Datasphere. The extraction uses Replication flow connected to CDS view with Initial and Delta mode.

In the delta phase we observed a bottleneck when data is transferred from Subscriber Logging table to Resilient Buffer table, specifically the transfer job takes long time to be completed and Subscriber Logging table's records grow instead of being flushed in the buffer immediately.

Additional information:

• Buffer Settings:
  • Maximum Buffer Records -> 500.000
  • Package Size -> 10.000
• Job Settings of CDC Engine:
  • HEALTH_CHECK_PERIOD -> 10
  • LOGTAB_RECORDS_LIMIT -> 2000000000
  • LOGTAB_RECORDS_LIMIT_WARNING -> 1500000000
  • OBSERVER_MAX_JOBS -> 1
  • OBSERVER_MIN_JOBS -> 1
  • OBSERVER_MIN_RUNTIME -> 60
  • OBSERVER_PERIOD -> 0
  • TRANSFER_MAX_JOBS -> 2
  • TRANSFER_MAX_RUNTIME_MIN -> 360
  • TRANSFER_MIN_JOBS -> 2
  • TRANSFER_MIN_RUNTIME_SEC -> 15
• Replication Flow settings:
  • Delta Load Interval -> 00 : 05
  • Object Thread Count for Delta Loads (1-10) -> 1
• Data volume
  • Delta load Approx. 25.000 records/hour

We attempted to mitigate the issue by reducing the Package Size to 10.000 and adjusting the TRANSFER_MIN_JOBS and TRANSFER_MAX_JOBS to 2 in order to commit smaller batches and take advantage of jobs parallelization, however the bottleneck persists.

Delta phase performance improvement

In the current setup, the buffer table remains empty while new packages are immediately marked as “In Process”. At the same time, the Subscriber Logging table still contains records waiting to be processed. This suggests that RMS is consuming data from the buffer faster than the source system is able to supply it.

How can we improve the performance of the replication flow during the delta phase, specifically by optimizing the step that pushes data from the Subscriber Logging table to the Resilient Buffer table?