liveBook · Manning

Keyed state

As explained in Kafka for Architects

Keyed state is Flink-managed state associated with specific record keys and accessed or updated during keyed processing.

Keyed state is Flink-managed state associated with a specific key. Events are grouped by key before a stateful operation accesses or updates that key’s state. Unlike Kafka partitioning, Flink’s partitioning distributes data logically among parallel tasks and does not depend on Kafka. section 8.4.2

Orders by region

To count orders per region, use the region as the key. Each region’s events are grouped so its count can be updated separately. section 8.4.2

State storage affects recovery. In-memory state with checkpoints held by the JobManager can recover from a TaskManager failure but is lost if the JobManager or cluster fails. For larger, longer-lived state, RocksDB can store state on disk, while checkpoints can be saved externally to HDFS or S3. section 8.4.2

How Kafka for Architects covers it

How deep it goes
Defines key association and discusses partitioning, storage, and recovery without detailing state-access APIs.
Examples
Uses orders grouped by region to illustrate a separate count for each key.
What you can do after
Evaluate/understand how keyed aggregation and state-storage choices affect recovery in Flink.
How it connects
Connects keyed state to stream partitioning, aggregation, operator state, checkpointing, and RocksDB.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.