Log compaction
Log compaction is a storage technique that retains the latest record for each key while removing superseded records.
Kafka for Architects
Log compaction is a Kafka storage process that removes older records for a key while retaining the record with the latest value and preserving record offsets.
- How it is used
- Uses compacted topics to preserve the latest keyed state for services that can reconstruct a current-state view.
- How deep it goes
- Explains cleaner threads, unchanged offsets, tombstone retention, and cleanup eligibility without a low-level implementation walkthrough.
- Examples
- Illustrates current-state storage with account balances and customer profiles.
- What you can do after
- Operate/configure it with cleanup.policy=compact and evaluate current-state storage against full-history audit needs.
Apache Kafka in Action
Log compaction is a Kafka cleanup technique that retains the latest record for each message key and removes obsolete records.
- How it is used
- Keeps current keyed data, such as customer addresses, without deleting it solely because of age.
- How deep it goes
- Explains keyed cleanup through commands, cleaner thresholds, segments and consumer offsets, while showing that removal is not immediate.
- Examples
- Uses changing customer addresses and console-produced product prices, then inspects partition logs with kafka-dump-log.sh.
- What you can do after
- Operate/configure it with cleanup.policy=compact and keyed producer records, then inspect why obsolete records may remain temporarily.
Kafka Streams in Action, Second Edition
Log compaction is Kafka's process of removing older records for a key while retaining the most recent record for that key.
- How it is used
- Restores the latest keyed state after crashes and supports compacted Kafka Streams changelogs.
- How deep it goes
- Explains per-key cleanup, background segment rewriting, policies, and tombstone deletion, while deferring broader compaction details.
- Examples
- Uses stock-price updates, duplicate-key logs, compacted changelogs, and a Materialized Java configuration.
- What you can do after
- Operate/configure compacted topics and Kafka Streams changelogs for recoverable keyed state.
Read how Kafka Streams in Action, Second Edition explains log compaction →
What to know first
- consistencyConsistency is the degree to which related data, behavior, or results agree with one another and conform to applicable rules or expectations.
- clientA client is a person, organization, device, or software component that receives or uses a product, service, resource, or functionality provided by another party or system.
- topicA topic is a named channel or category that groups messages or records for publication and delivery to interested consumers.
- log segmentA log segment is a file-backed portion of a Kafka partition log containing a contiguous range of records and associated indexes.
Related concepts
- log retentionalternative to · Log retention is a policy governing how long log records are kept and when they are removed.
- partition keyused by · A partition key is a value or attribute used to determine the partition to which a data item, record, or message is assigned or routed.
- log cleanerimplemented by · The Kafka log cleaner is a broker subsystem that compacts partition logs by removing obsolete records according to log-compaction rules.
- topicconfigures · A topic is a named channel or category that groups messages or records for publication and delivery to interested consumers.
Questions about log compaction
What is log compaction?
Log compaction is a storage technique that retains the latest record for each key while removing superseded records.
Which Manning books explain log compaction?
Kafka for Architects, Apache Kafka in Action and Kafka Streams in Action, Second Edition.
What should I know before learning log compaction?
Manning books that teach log compaction usually introduce consistency, client, topic and log segment first.
What is the difference between log compaction and log retention?
Log compaction is a storage technique that retains the latest record for each key while removing superseded records. Log retention is a policy governing how long log records are kept and when they are removed.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


