liveBook · Manning

State store

As explained in Kafka for Architects

A local state store is a stateful storage mechanism located on the same node as processing logic for low-latency access to application state.

An order-enrichment service needs current customer information whenever an order arrives. A local state store keeps that information alongside the processing logic, so the service can look it up without making a remote request for every order. In Kafka Streams, state stores also hold intermediate results for stateful operations such as joins and aggregations. section 8.1 section 8.3.3

Counting orders by region

A counting service uses the region as a key and its order count as the stored value. Keying incoming orders by region routes orders for that region to the same processing instance, where the counter can be updated locally. The store is a key-value view of current state, rather than another stream of individual orders. section 8.3.2

A Kafka Streams task combines an assignment of input partitions, a processing topology, and the state stores that topology needs. Each task maintains its own local state; tasks do not exchange historical events directly to perform joins or aggregations. Kafka Streams backs up store changes in a changelog topic and restores state when a failed task is reassigned. Store objects therefore need suitable serializers and deserializers. section 8.3.4 section 8.3.2

A local store can also serve interactive queries when it implements the required queryable-store interface. Code on the same instance can read it directly. For a key held elsewhere, Kafka Streams supplies key-to-instance metadata, but the application must provide the endpoint and remote-query logic. Local storage is not, by itself, a cluster-wide query service. section 8.3.8

Figure 8.16 When a client requests the state for a given key, the instance first checks its locally cached metadata. If the state resides on another instance, the metadata identifies the correct instance to forward the query to.
A screenshot of a computer

Description automatically generated

The query-routing diagram distinguishes a local store lookup from forwarding a request to the instance that owns the key. section 8.3.8

Low-latency access still has a resource cost. RocksDB stores data primarily on disk and caches frequently accessed data in memory; its caches and write buffers, input-record buffering, and changelog topics all need capacity planning. Restoring a large store can strain disk and memory and delay readiness. Partition planning, RocksDB tuning, standby tasks, and restoration benchmarks help assess that trade-off. section 8.5.1 section 8.5.5

How Kafka for Architects covers it

How deep it goes
Explains keyed storage, task ownership, changelog recovery, and query routing, while leaving remote-query implementation to the application.
Examples
Uses customer-data lookups for order enrichment and per-region counters for order aggregation.
What it assumes
Assumes familiarity with Kafka events and partitions while defining state stores through keyed counters and their recovery.
What you can do after
Operate/configure it by planning storage and restoration capacity and distinguishing local queries from application-implemented remote queries.
How it connects
Connects state stores to Kafka partitions, processing topologies, changelog topics, RocksDB, and interactive queries.

Also explained in

Compare how 3 books explain state store →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.