A Kafka message includes a timestamp alongside its key, value, and other metadata. For time-based retention, that timestamp helps determine when the message is old enough to remove. The timestamp source is therefore an architectural choice, not just a producer detail. section 2.4.1 section 2.4.5
Two timestamp sources have different operational consequences:
- CreateTime is set by the producer. An inaccurate producer clock can cause premature deletion or unnecessarily long retention. Brokers can reject producer timestamps outside ranges set by log.message.timestamp.after.max.ms and log.message.timestamp.before.max.ms. section 2.4.5 section 3.2.6
- LogAppendTime is assigned when the broker receives the message, making deletion more predictable. When messages are batched and compressed, however, a single LogAppendTime timestamp can represent the whole batch rather than each message’s individual time. section 2.4.5 section 3.2.6
Plan for a disconnected consumer
If a consumer may be disconnected for one day, retain its messages for longer than a day, with a margin. Using LogAppendTime gives retention a broker-assigned reference point; keep the client’s original time in the payload when that time matters to the business. section 2.4.5
Do not confuse a message timestamp with event time. A transaction may carry a business timestamp in its payload, while Kafka Streams ordinarily processes records in partition-offset order; timestamps can appear out of order because of producer delays or clock differences. To base Kafka Streams logic on embedded event time, implement a TimestampExtractor. Agree on timestamp sources across systems so retention and processing use the intended meaning of time. section 3.2.2 section 8.5.3 section 3.2.6
