A stream processor may keep the latest value or an intermediate count for each key in a local state store. If its instance fails, that local state must be restored. Kafka Streams records state updates in an internal changelog topic, usually compacted, so a replacement instance can rebuild the store. section 12.4.1 section 12.6
A count that survives failure
Suppose keyed records arrive as (K1, 2), (K2, 1), and (K2, 2). A count() aggregation leaves K1 at 1 and K2 at 2. Each update changes the table's state and is written to its changelog topic. If another instance takes over the task, it reads the changelog to populate a new state store rather than starting the counts over. section 12.4.2 section 12.4.1 section 12.6
Figure 12.8 To count the number of messages per key, a state store with the current count per key is needed. The changes to the state store are stored in a compacted changelog topic.
Follow the current count per key in the state store and its updates in the compacted changelog topic. section 12.4.2
Recovery by rebuilding can take work. Setting num.standby.replicas=1 or higher keeps a backup state store on another instance, allowing it to take over without rebuilding from scratch. A changelog topic serves state recovery; a repartition topic has a different job: routing records by key before operations such as joins and aggregations. section 12.6
