A Kafka Streams state store is local storage attached to a stream processor. Kafka Streams creates one state store for each processor in each task, allowing processing state to remain associated with the task handling a partition. section 12.6
State stores shape both scaling and recovery:
- Tasks are distributed across threads and application instances, and each task has its own state store. section 12.6
- When an instance fails, another instance rebuilds the missing store by reading its internal changelog topic. section 12.6
- Setting num.standby.replicas to one or higher keeps backup stores on another instance, allowing takeover without rebuilding from scratch. section 12.6
- State-store metrics expose implementation details such as RocksDB memory-table entry counts and sizes. section 15.3.4
Failure and takeover
Imagine nine tasks spread across two application instances with multiple processing threads. If one instance fails, another takes over its tasks and repopulates each task's state store from the corresponding changelog. Configuring a standby replica lets the replacement use an already maintained copy instead. section 12.6
Figure 12.13 Kafka Streams applications are divided into tasks that are distributed across different threads on different instances. If an instance fails, another instance takes over the tasks.
Trace how tasks and their state stores are distributed across threads and instances, then see where failure recovery transfers work. section 12.6
