Suppose an application must count orders by region as they arrive. Filtering an order can be done in isolation; counting requires a counter carried forward from earlier orders. That retained context makes the operation stateful. The design must keep orders for a region with the instance that owns its counter, while making the counter durable and quick to access. section 8.3.2
Kafka Streams uses the event key to route related orders to the same partition and processing instance. A state store maintains a value for each key—for example, a counter for each region. Kafka Streams backs local state with a Kafka changelog topic so an instance can restore it after failure. State-store objects also need suitable serializers and deserializers. section 8.3.2
Statefulness changes both the processing model and the operating plan:
- Use aggregations, joins, or time windows when a result depends on multiple events. An hourly order count answers a different question from a count over the entire stream. section 8.1.3 section 8.0
- Plan for restoration. Rebuilding a store can strain disk and memory and delay readiness; partition planning, RocksDB tuning, standby tasks, and restoration benchmarks can reduce the impact. section 8.5.5
- Do not assume state scales like a stateless service. Adding a Kafka broker does not automatically redistribute its partitions and data; moving them consumes resources and can affect latency. section 9.2.1
Choosing a processing home
For a customer view built from transactions and profiles, an intermediate streaming service can perform joins and publish the aggregated result to Kafka. A short-lived serverless function is a less natural fit for stateful joins or aggregates: it needs an external state store, adding latency and consistency concerns. Long-running services or stream-processing frameworks are better suited to that workload. section 8.0 section 12.2.2
Stateful processing is useful when results must reflect accumulated events, especially when low-latency updates matter. If once-a-day processing is sufficient, batch processing may be simpler and more cost-effective. The architectural choice is therefore not just how to compute a result, but where its state lives, how it recovers, and whether the required latency justifies those costs. section 5.1.4 section 8.3.2
