A compacted topic retains the latest record for each key instead of deleting complete log segments by time or size. This preserves current state for applications recovering after a crash or restart. section 2.6.1 section 2.6.2
The implementation model is:
- A background log cleaner recopies log segments and removes an older record when a later record has the same key. section 2.6.2
- Enable compaction for a topic with the cleanup.policy=compact property. section 2.6.2
- Delete a key through compaction by writing a null value, which creates a tombstone marker. section 2.6.2
- Combine compact and delete policies when a large key space could otherwise allow log segments to keep growing. section 7.4.7
Recovering stock prices
Use a stock ticker symbol as the key and its current price as the value. After a crash or restart, the application can rebuild the latest known price for every ticker without replaying every historical update. section 2.6.2
Figure 2.7 On the left is a log before compaction. You’ll notice duplicate keys with different values. These duplicates are updates. On the right is after compaction, which retains the latest value for each key, but it’s smaller in size.
Compare duplicate key updates before compaction with the smaller log retaining only each key’s latest value. section 2.6.2
