Sensor readings, program logs, and short-lived messages may not need to stay in Kafka indefinitely. Log retention removes data by age or partition size, keeping storage from growing without bound. Unlike log compaction, it does not inspect message keys to decide which records still matter. section 10.2 section 10.3.1
Retention deletes whole segments, not individual old records. Under time-based retention, a segment qualifies when its newest message is older than the retention period. A size limit instead triggers deletion of the oldest segment when a partition exceeds the configured size. The log cleaner checks for outdated data periodically. section 10.3.1 section 10.2
Set the limits to match how long the data remains useful:
- Use `retention.ms` for a topic’s retention period. The stated default is seven days. section 10.3.1
- Use `retention.bytes` to limit partition size. Its default, `-1`, disables size-based retention. section 10.3.1
- Use `log.retention.ms` and `log.retention.bytes` to adjust cluster defaults; time and size limits can be combined. section 10.3.1
Try a one-minute retention period
Create a topic, produce several messages, then set its `retention.ms` to `60000` with `kafka-configs.sh`. Once the newest message in a segment is older than 60 seconds, the segment can be deleted. A partition still needs a segment, so a new one appears at the current offset. If the files have not changed yet, check again later or inspect another partition; old files may briefly have a `.deleted` extension. section 10.3 section 10.3.1
$ kafka-configs.sh --alter \
--topic products.prices-retention \
--add-config retention.ms=60000 \
--bootstrap-server localhost:9092
Completed updating config for topic products.prices-retention.
The command sets `retention.ms=60000` on a topic without changing the cluster-wide default. section 10.3.1
Choose retention when old data can safely disappear regardless of its key—for example, time-limited logs or data subject to deletion requirements. It is simpler and places less overhead on brokers than compaction, but it can also delete a still-important record merely because it is old. For keyed data whose latest value must remain available, consider log compaction; the two cleanup methods can also be combined. section 10.2
