A retention policy controls how long Kafka keeps records before deleting them. Kafka can trigger cleanup by elapsed time, partition size, or both. Time-based retention uses retention.ms; size-based retention uses retention.bytes. section 10.3.1
Kafka deletes complete log segments rather than individual records. With the default time-based behavior, a segment becomes eligible when its newest record is older than seven days. After deletion, the partition keeps at least one segment and creates a new one when needed. section 10.3.1
Choose retention settings according to operational needs:
- Set retention.ms when records must remain available for a defined duration, such as 60 seconds or several days. section 10.3.1
- Set retention.bytes to limit partition growth; its default value of -1 disables size-based retention. section 10.3.1
- Retain data long enough for interested systems to retrieve it, then consider tiered storage when historical data is costly to keep in Kafka. section 18.4.2
- Expect short retention to let partition sizes rebalance after partition changes, while old partitioning and message-order effects persist until retained data expires. section 6.1.3
Set a 60-second retention period
Run kafka-configs.sh to change the products.prices-retention topic: kafka-configs.sh --alter --topic products.prices-retention --add-config retention.ms=60000 --bootstrap-server localhost:9092. After the segment's newest record is older than 60 seconds, Kafka deletes the segment; inspecting the partition directory shows the newly active segment and checkpoint files. section 10.3.1
Figure 10.2 Log retention checks whether a segment is larger than 1 GB or whether the newest message is older than seven days. If so, the whole segment is deleted.
Compare the time- and size-based checks that make a complete segment eligible for deletion. section 10.3.1
