Log compaction
Log compaction is a storage technique that retains the latest record for each key while removing superseded records.
Kafka for Architects
Log compaction is a Kafka storage process that removes older records for a key while retaining the record with the latest value and preserving record offsets.
- How it is used
- Uses compacted topics to preserve the latest keyed state for services that can reconstruct a current-state view.
- How deep it goes
- Explains cleaner threads, unchanged offsets, tombstone retention, and cleanup eligibility without a low-level implementation walkthrough.
- Examples
- Illustrates current-state storage with account balances and customer profiles.
- What you can do after
- Operate/configure it with cleanup.policy=compact and evaluate current-state storage against full-history audit needs.
Apache Kafka in Action
Log compaction is a Kafka cleanup technique that retains the latest record for each message key and removes obsolete records.
- How it is used
- Keeps current keyed data, such as customer addresses, without deleting it solely because of age.
- How deep it goes
- Explains keyed cleanup through commands, cleaner thresholds, segments and consumer offsets, while showing that removal is not immediate.
- Examples
- Uses changing customer addresses and console-produced product prices, then inspects partition logs with kafka-dump-log.sh.
- What you can do after
- Operate/configure it with cleanup.policy=compact and keyed producer records, then inspect why obsolete records may remain temporarily.
Kafka Streams in Action, Second Edition
Log compaction is Kafka's process of removing older records for a key while retaining the most recent record for that key.
- How it is used
- Restores the latest keyed state after crashes and supports compacted Kafka Streams changelogs.
- How deep it goes
- Explains per-key cleanup, background segment rewriting, policies, and tombstone deletion, while deferring broader compaction details.
- Examples
- Uses stock-price updates, duplicate-key logs, compacted changelogs, and a Materialized Java configuration.
- What you can do after
- Operate/configure compacted topics and Kafka Streams changelogs for recoverable keyed state.
Read how Kafka Streams in Action, Second Edition explains log compaction →
Related concepts
- partitioning keyused by · A partitioning key is a value or attribute used to determine which partition stores or receives a record or data item.
- log retentionalternative to · Log retention is a policy governing how long log records are kept and when they are removed.
- log cleanerimplemented by · The Kafka log cleaner is a broker subsystem that compacts partition logs by removing obsolete records according to log-compaction rules.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.