A compacted topic retains at least the latest record for each unique, non-null key. It suits state that must remain available after older updates stop mattering, such as configuration, changelogs, profiles, or current prices. section 2.0 section 2.5
Unlike an ordinary time- or size-retained streaming topic, compaction preserves current state rather than a complete history. It is configured at the topic level, so producers and consumers need no API changes. section 2.5.1 section 2.5.4
Design and operational constraints include:
- Require non-null keys; route each key consistently to the same partition, making key-based partitioning mandatory. section 2.5.1
- Retain older records temporarily because cleanup is asynchronous and performed periodically by Kafka’s log cleaner. section 2.5.1 section 2.5.4
- Preserve sequential log access rather than providing random lookup by key; consumers may encounter multiple versions of one key. section 2.5.1 section 4.2.12
Account balances for a dashboard
An AccountService publishes each new balance using the account number as the key. A DashboardService can replay the topic from the beginning and rebuild current balances in memory, without needing every historical update. section 2.5.1
Figure 2.15 A topic where only the final message is important for the consumer. The DashboardService is only interested in the latest balances, not their provenance or history.
Shows why a dashboard needs only the final balance for each account rather than the complete provenance of every update. section 2.5.1
Use compaction when current keyed state is more valuable than complete history. Validate the choice with repeated updates and deletion records, then monitor client behavior, latency, broker performance, and disk usage. section 2.5.4
