The Kafka log cleaner compacts partition logs by removing outdated records while preserving the latest record for each key. Cleanup is asynchronous: the cleaner checks periodically rather than processing every new message immediately. section 10.4.1 section 10.4.2
A partition becomes eligible when its dirty-segment ratio reaches the configured threshold. The default ratio is 0.5; lowering it reduces stale-storage overhead but increases resource usage because compaction reads the entire partition. section 10.4.1
During compaction, the cleaner:
- Finds the latest offset for each key in uncompacted segments. section 10.4.2
- Reads older messages and omits entries superseded by newer values. section 10.4.2
- Writes retained records into a new segment, then replaces the old segments when the new segment is full. section 10.4.2
- Leaves the newest segment uncompacted, so it may contain multiple records for one key. section 10.4.2
- Preserves offsets and message order while removing outdated entries. section 10.4.2
Compacting repeated prices
Producing energy drink:5 followed by energy drink:6 leaves the latest value in a compacted segment, while the newest uncompacted segment can temporarily contain both records. kafka-dump-log.sh can inspect these segments and verify the retained offset.
