Fault tolerance in Smart Data Streaming

[former_member258450]

This is another comparison with Apache Kafka where Kafka replicates the messages in a distributed manner among the hosts it is running on. Which means that even if messages are lost, they are always recoverable as another host(hosts*) holds the data.

I was wondering how this happens in the case of HANA Smart Data Streaming. Are the messages stored locally on the streaming server AND in HANA as well? What happens if the streaming node is down for some reason? Are the messages lost? How is fault tolerance managed/ guaranteed in SAP Smart Data Streaming?

[JWootton · Accepted solution]

See my answer to your other question comparing message handling of Kafka and Smart Data Streaming (SDS) - the key point being that these are complementary and not alternative ways of solving the same problem. Kafka is a message broker for delivering messages unchanged. SDS is a streaming analytics engine.

With that said, I'm happy to talk about fault tolerance in SDS:

- When an SDS project is configured for guaranteed delivery, messages are held in "event windows" which are held in-memory within the SDS project itself, but are backed by a disk-based log store and are thus recoverable. The reason guaranteed delivery is an option is that it's not needed in all cases and there is overhead. It can be configured at the individual stream level and doesn't have to be set for all streams in a project. See this section of the SDS Developers Guide for more detail.

- SDS can be configured as a multi-node cluster with auto-failover for projects. The SDS cluster managers (peer network - no single point of failure) monitors all projects via a heart beat. A project that goes down will be restarted on an available cluster node and all windows will be restored from disk.

- For full guaranteed delivery, SDS publishers will of course need to watch for confirmation that messages sent to SDS have been committed, and be prepared to re-send in the case that the message wasn't received and committed.

[former_member258450 · Accepted solution]

Thank you, Jeff. Very detailed explanation, indeed.