liveBook · Manning

Checkpointing

As explained in Kafka for Architects

Checkpointing is the periodic capture and persistence of streaming application state so it can be restored after failure.

A streaming job counting orders by region must retain each region’s running total. In Flink, that total is keyed state. A source operator may also retain its reading progress as operator state. Checkpointing periodically captures application state so the job can recover after a failure. section 8.4.2

Where checkpoints live determines what failures they can survive. Checkpoints held in JobManager memory can support recovery from a TaskManager failure, but are lost if the JobManager or cluster fails. Production jobs can save checkpoints to external storage such as HDFS or S3. RocksDB can hold large, long-lived Flink state on disk and work with checkpointing; Kafka Streams instead uses Kafka for changelogs and checkpoints. section 8.4.2

How Kafka for Architects covers it

How deep it goes
Contrasts memory-held checkpoints with external storage and their failure implications, without detailing configuration or recovery steps.
Examples
Uses orders grouped by region and a source operator’s reading progress to illustrate state worth retaining.
What it assumes
Introduces keyed and operator state while assuming familiarity with streaming applications.
What you can do after
Operate/configure it by selecting durable external checkpoint storage instead of relying on JobManager memory for production recovery.
How it connects
Links checkpointing to keyed state, operator state, RocksDB, HDFS, S3, and Kafka Streams changelogs.

Also explained in

Compare how 6 books explain checkpointing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.