Checkpointing
Checkpointing is the recording of an intermediate state or progress point in a process to support recovery, resumption, inspection, or control.
Architecting an Apache Iceberg Lakehouse
A checkpoint is persisted progress information from which an incremental data-processing workflow can resume after interruption.
- How it is used
- Supports resumable, fault-tolerant ingestion across micro-batch and continuous streaming workflows.
- How deep it goes
- Explains offsets, state, retries, and transactional coordination, while deferring implementation-specific configuration and recovery tuning.
- Examples
- Uses Spark Kafka-to-Iceberg streaming, Flink checkpointed Kafka ingestion, and five-minute micro-batch progress boundaries.
- What you can do after
- Operate and recover incremental ingestion pipelines from persisted progress while limiting duplicates, skipped records, and partial commits.
Read how Architecting an Apache Iceberg Lakehouse explains checkpointing →
Kafka for Architects
Checkpointing is the periodic capture and persistence of streaming application state so it can be restored after failure.
- How it is used
- Protects stateful Flink streaming jobs against failures by periodically capturing application state.
- How deep it goes
- Contrasts memory-held checkpoints with external storage and their failure implications, without detailing configuration or recovery steps.
- Examples
- Uses orders grouped by region and a source operator’s reading progress to illustrate state worth retaining.
- What you can do after
- Operate/configure it by selecting durable external checkpoint storage instead of relying on JobManager memory for production recovery.
ScyllaDB in Action
Checkpointing records completed progress so that a process can pause and resume from an earlier point without restarting.
- How it is used
- Supports pausing and resuming bulk data migrations without repeating completed work.
- How deep it goes
- Explains the operational rationale and a simple storage approach, but stops short of defining a sophisticated checkpoint protocol.
- Examples
- Uses Spark Migrator progress tracking and custom migrator tables storing completed token ranges.
- What you can do after
- Operate a bulk migration that pauses and resumes from recorded progress.
AI Agents and Applications
- How it is used
- Adds short-term conversational memory to LangGraph agents as part of making them more production-ready.
- How deep it goes
- Defines snapshots at graph super-steps and implements conversational persistence, while noting but not implementing recovery and human-approval workflows.
- Examples
- Uses a travel assistant, shared thread IDs, and an in-memory checkpointer to demonstrate conversational continuity.
- What you can do after
- Apply it via LangGraph to preserve conversational state across turns using a thread ID and checkpointer.
Read how AI Agents and Applications explains checkpointing →
MongoDB 8.0 in Action, Third Edition
A stream processing checkpoint is a persisted record of processing progress and state that allows a stream processor to resume consistently after interruption.
- How it is used
- Maintains resumable progress and state for stream processors after interruption.
- How deep it goes
- Explains checkpoint records, commitment, restart recovery, and durable storage, but does not specify checkpoint configuration commands.
- Examples
- Uses checkpoint documents, stateful operators, processor restarts, WiredTiger data files, and journal replay as concrete examples.
- What you can do after
- Operate Atlas Stream Processing with resumable progress and understand how WiredTiger combines checkpoints with journaling for recovery.
Read how MongoDB 8.0 in Action, Third Edition explains checkpointing →
Designing AI Agents
The practice of recording execution state at checkpoints so a workflow can resume after interruption.
- How it is used
- Preserves resumable agent work across resets and handoffs, and supports rollback during repair attempts.
- How deep it goes
- Details checkpoint contents, local persistence, recovery verification, replay risks, and repair rollback, while deferring stronger distributed-store guarantees.
- Examples
- Uses a progress log and JSON feature list, a local atomic checkpoint, and a workspace snapshot before repair.
- What you can do after
- Apply checkpointing to resumable workflows and rollback attempts while accounting for repeated tasks and external effects.
Building Reliable AI Systems
The practice of saving execution state so a process can be paused and later resumed.
- How it is used
- Supports agent state management by allowing execution to pause, resume, or recover after sandbox failure.
- How deep it goes
- Connects checkpointing to pause, resume, and crash recovery, but does not detail checkpoint formats or save-and-restore procedures.
- Examples
- Uses a sandbox crash as the recovery case: restore harness state and continue from the last checkpoint.
- What you can do after
- Understand how an agent harness can recover execution after a sandbox crash using the last checkpoint.
Read how Building Reliable AI Systems explains checkpointing →
Also covered in
- Local Image GenerationIn early access (MEAP)
- 100 SQL Server Mistakes and How to Avoid Them
- Writing for Developers: Blogs that get read
- Distributed Machine Learning Patterns
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
- Apache Kafka in Action
- Designing Deep Learning Systems: A guide for software engineers
- Kafka Streams in Action, Second Edition
- Grokking AI Algorithms, Second Edition
- LLMs in Production
- Deep Learning with PyTorch, Second Edition
- Context EngineeringIn early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- memoryMemory is a storage capability or mechanism for retaining information or computational state for later access or use.
- dataset splitA dataset split is the division of a dataset into distinct subsets assigned different purposes, such as training, validation, or evaluation.
- replicationReplication is the creation or maintenance of multiple copies of data, state, or functionality across distinct systems, nodes, or locations.
Related concepts
- trainingproduced by · Training is the process of fitting a machine-learning model to data by adjusting its parameters or learned representations to improve its performance on a task.
- checkpoint mergingproduced by · Checkpoint merging is the technique of combining parameters from multiple model checkpoints according to specified proportions to create a new model.
Questions about checkpointing
What is checkpointing?
Checkpointing is the recording of an intermediate state or progress point in a process to support recovery, resumption, inspection, or control.
Which Manning books explain checkpointing?
Architecting an Apache Iceberg Lakehouse, Kafka for Architects, ScyllaDB in Action, AI Agents and Applications, MongoDB 8.0 in Action, Third Edition, Designing AI Agents and Building Reliable AI Systems.
What should I know before learning checkpointing?
Manning books that teach checkpointing usually introduce language model, memory, dataset split and replication first.
What is the difference between checkpointing and exactly-once side effect?
Checkpointing is the recording of an intermediate state or progress point in a process to support recovery, resumption, inspection, or control. A guarantee that an external effect occurs once, even when an operation may be retried.
What kinds of checkpointing are there?
Kinds of checkpointing include asynchronous checkpointing.
What is checkpointing also called?
Checkpointing is also called checkpoint.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.






