Kafka Consumer Duplicate Processing in SAP Integration Suite

[Marielo]

Hi everyone,

I'm investigating a Kafka consumer behavior in SAP Integration Suite where the same message is sporadically processed by different BTP runtime instances.

I was able to reproduce the behavior in our DEV tenant using a simple iFlow.

Configuration / Default:

Kafka Partitions: 1

Parallel Consumers: 1

Error Handling: Skip Failed Message

Heartbeat Interval: 3 seconds

Session Timeout: 10 seconds

Request Timeout: 30 seconds

Why did I add a 2-minute delay?

In the normal flow, message processing is very fast, so the duplicate behavior is difficult to reproduce consistently. I added an intentional 2-minute delay to keep the message processing active longer and determine whether a long-running execution, consumer session timeout, or possible rebalance could be related to the duplicate processing.

Most messages completed successfully after the 2-minute delay. However, under load, some messages were processed again by different runtime instances while the original execution was still running.

For example:

Run 1 - 08:16:02 - Instance 1 - Completed

Run 2 - 08:16:14 - Instance 0 - Completed

Run 3 - 08:16:26 - Instance 4 - Completed

The executions started approximately 12 seconds apart, while the configured Session Timeout is 10 seconds.

There were no application errors, no Retry Failed Message configuration, and all executions completed successfully.

My current hypothesis is that this could be related to a consumer rebalance or redelivery before the Kafka offset is committed, but I cannot confirm this from the MPL alone.

Has anyone observed similar behavior?

I'm particularly interested in understanding:

1. When does the Kafka Sender Adapter commit the offset relative to the iFlow completion?

2. Can a rebalance cause another runtime instance to process the same Kafka record while the original execution is still running?

3. Could the approximately 12-second interval between duplicate executions be related to the configured 10-second Session Timeout? If so, why did most messages with the same intentional 2-minute processing delay complete successfully without triggering a timeout, rebalance, or duplicate execution, while only some messages were processed again by another runtime instance?

4. What is the recommended best practice to prevent duplicate processing in this scenario?

Thanks in advance for any insights.

[Marielo]

Hi everyone,

I'm investigating a Kafka consumer behavior in SAP Integration Suite where the same message is sporadically processed by different BTP runtime instances.

I was able to reproduce the behavior in our DEV tenant using a simple iFlow.

Configuration / Default:

Kafka Partitions: 1

Parallel Consumers: 1

Error Handling: Skip Failed Message

Heartbeat Interval: 3 seconds

Session Timeout: 10 seconds

Request Timeout: 30 seconds

Why did I add a 2-minute delay?

In the normal flow, message processing is very fast, so the duplicate behavior is difficult to reproduce consistently. I added an intentional 2-minute delay to keep the message processing active longer and determine whether a long-running execution, consumer session timeout, or possible rebalance could be related to the duplicate processing.

Most messages completed successfully after the 2-minute delay. However, under load, some messages were processed again by different runtime instances while the original execution was still running.

For example:

Run 1 - 08:16:02 - Instance 1 - Completed

Run 2 - 08:16:14 - Instance 0 - Completed

Run 3 - 08:16:26 - Instance 4 - Completed

The executions started approximately 12 seconds apart, while the configured Session Timeout is 10 seconds.

There were no application errors, no Retry Failed Message configuration, and all executions completed successfully.

My current hypothesis is that this could be related to a consumer rebalance or redelivery before the Kafka offset is committed, but I cannot confirm this from the MPL alone.

Has anyone observed similar behavior?

I'm particularly interested in understanding:

1. When does the Kafka Sender Adapter commit the offset relative to the iFlow completion?

2. Can a rebalance cause another runtime instance to process the same Kafka record while the original execution is still running?

3. Could the approximately 12-second interval between duplicate executions be related to the configured 10-second Session Timeout? If so, why did most messages with the same intentional 2-minute processing delay complete successfully without triggering a timeout, rebalance, or duplicate execution, while only some messages were processed again by another runtime instance?

4. What is the recommended best practice to prevent duplicate processing in this scenario?

Thanks in advance for any insights.