A Kafka retention policy determines how long records remain available in a topic. Records are appended to a log and cannot be individually edited or removed. Consumers can replay retained records, but retention is not tied to whether they have read them: expired data can be deleted even if it remains unprocessed. section 1.4.3 section 2.4.5
Choose a policy according to what consumers and the business need:
- Time-based retention keeps data for at least the configured period; seven days is the default. Allow more time than the longest expected consumer disconnection, with a margin. Deletion after the period is not immediate. section 2.4.5
- Size-based retention preserves a configured amount of recent data. A surge in event volume can shorten the available history just when a consumer is falling behind, making time-based retention safer for critical data. section 2.4.5
- Compaction serves a different need: a compacted topic retains at least the latest record for each key. Choose between compaction and delete retention by asking whether the requirement is current state, historical access, auditability, or reprocessing. section 2.0 section 10.1.3
For time-based retention, the message timestamp matters. CreateTime comes from the producer; LogAppendTime is assigned when the broker receives the record and can make deletion more predictable. If CreateTime is used, broker timestamp limits can reject records outside the accepted range. section 2.4.5
A five-day topic policy
A dashboard team chooses at least five days of retention after considering consumer needs and the possibility that development and production require different settings. The topic configuration uses retention.ms: 432000000. That value is a design decision to revisit against consumer availability, not a promise that records disappear exactly five days after arrival. section 2.4.7 section 2.4.5
asyncapi: 3.0.0
info:
title: Customer 360 Project
version: 1.0.0
description: >
This project aims to aggregate data from various sources, providing a
comprehensive view of customer information.
channels:
customerProfile:
description: Storing profile information of customers.
address: CUSTOMER_PROFILES
messages:
profile:
bindings:
kafka:
key:
type: string
description: identifier of the customer
bindings:
kafka:
partitions: 9
replicas: 3
topicConfiguration:
retention.ms: 432000000 #1
min.insync.replicas: 2
The AsyncAPI channel shows retention.ms alongside partition, replication, and minimum in-sync replica settings, making retention an explicit part of topic design. section 2.4.7
Plan disk capacity and monitor usage when setting retention. Unlike acknowledgment-based removal, Kafka's configured period can end while a client is disconnected or still processing; the retention window must fit the client's nonfunctional requirements. section 5.2.5
