A balance dashboard needs each account’s current balance, not every balance change. Log compaction lets a Kafka topic retain at least the latest record for each key, so a consumer can rebuild current state after restarting. Set the topic’s cleanup.policy to compact; producers and consumers do not need a different API. section 2.5.1 section 4.2.12
The key and the cleanup process determine what remains:
- Use a non-null key that identifies the state being maintained, such as an account number. Records with the same key must reach the same partition; a poor key choice can also create uneven partition load. section 2.4.1 section 2.5.1
- Background cleaner threads rewrite old, closed log segments. They remove superseded records without reordering surviving records or changing their offsets. Cleanup is not immediate, so consumers may still encounter older values. section 2.5.2 section 4.2.12
- A null value acts as a deletion marker, or tombstone. Tombstones remain available for a configured period so consumers have time to observe deletions before cleanup removes them. section 2.5.2
A compacted topic is still a sequential commit log, not a key-value lookup service. To reconstruct state, read through the topic and keep the latest value seen for each key. The retained records do not provide a complete history of changes. section 2.5.1
Customer profiles as current state
A customer-profile topic can retain the latest profile for each customer ID. A service can rebuild an in-memory view from Kafka instead of relying on its own durable database. If the business must audit changes or return to earlier states, retaining the full event history is a different design choice. section 2.5.4 section 5.1.1
Compaction cannot be triggered explicitly. Cleaner eligibility depends on broker settings for the dirty-data ratio and minimum or maximum compaction lag. Test repeated updates and tombstones while monitoring consumer behavior, latency, server performance, and disk usage before treating a compacted topic as a durable source of current state. section 2.5.3 section 2.5.4
Figure 2.16 During the compaction process, the data within a topic is rewritten. For each key, only the latest value is retained, and messages where the value is null are deleted.
The before-and-after view makes the removal of superseded balances and deleted keys concrete. section 2.5.2
