Suppose a Kafka topic holds customer addresses. Each change produces a new record with that customer’s key. The old address is no longer useful, but deleting records merely because they are old could also remove the current address. Log compaction addresses this by removing an older record when a newer record has the same key. section 10.2
Keys are essential: a compacted topic rejects records without them. Records with the same key go to the same partition. Unlike age- or size-based log retention, compaction selects obsolete records by key; the two cleanup policies can also be combined. section 10.4 section 10.2
$ kafka-topics.sh \
--create \
--topic products.prices-compaction \
--partitions 3 \
--replication-factor 3 \
--config cleanup.policy=compact \
--bootstrap-server localhost:9092
Created topic products.prices-compaction.
Create the example topic with --config cleanup.policy=compact; topics typically use a retention policy by default. section 10.4
Update a product price
Produce energy drink:5, then energy drink:3, using parse.key=true and key.separator=: so “energy drink” is the key. Both records initially appear in the same partition; the older price does not disappear immediately. Sending a value without a key instead produces an InvalidRecordException. section 10.4
$ kafka-console-producer.sh \
--topic products.prices-compaction \
--property parse.key=true \
--property key.separator=: \
--bootstrap-server localhost:9092
>cola:2
>coffee pads:10
[…]
>energy drink:5
The producer command shows how to enter key:value records for a compacted topic. section 10.4
Compaction is periodic, not immediate. The log cleaner checks for eligible work and uses settings such as min.cleanable.dirty.ratio to decide when to act. Lowering that threshold can reduce space occupied by obsolete records, but cleaning is resource-intensive. Minimum and maximum compaction-lag settings provide further timing controls. section 10.4.1
The newest segment may still contain several records for one key; a cleaned segment retains at most one. Cleaning preserves the offsets and order of surviving records. If a consumer requests an offset whose record was removed, Kafka returns the next available message. Tombstone messages support selective deletion and are eventually removed according to their retention period. section 10.4.2 section 10.5
