Checkpointing persists processing state so an incremental ingestion workflow can resume from its last known good position after interruption. The state may include offsets, timestamps, change identifiers, and coordinated commit status. Reliable recovery depends on replayable sources, because non-replayable sources limit downstream delivery guarantees. section 6.1.2 section 6.2.3
A checkpoint supports ingestion reliability by:
- Recording which input records or offsets have been successfully committed, so failures retry only unfinished work. section 6.1.2
- Coordinating processing state, offsets, and transactional commits so a checkpoint is committed completely or not at all. section 6.1.2
- Giving micro-batch jobs explicit progress boundaries instead of ambiguous time ranges such as “yesterday.” section 6.2.2
- Supporting stateful operations such as deduplication, sessionization, and change data capture across continuously arriving events. section 6.2.3
- Requiring operational controls around commit frequency, state recovery, file generation, metadata growth, and compaction at high throughput. section 6.5.2 section 6.5.3
Spark Structured Streaming checkpoint
A Kafka-to-Iceberg Spark stream writes its recovery state to a durable checkpoint location: parsed.writeStream.format("iceberg").outputMode("append").option("checkpointLocation", "s3a://checkpoints/my-table/").start("my_catalog.db.my_table"). With an appropriate replayable source and processing configuration, the stream can resume while supporting exactly-once semantics. section 6.4.1
Figure 6.3 By using checkpointing to track which incoming records have been committed successfully, the system knows which records to retry when a failure occurs.
The figure traces how checkpointed commit state identifies successfully committed records and determines which records to retry after failure. section 6.1.2
