A Kafka message has a timestamp in its technical metadata, alongside its optional key and headers and its value. That timestamp is not necessarily when the event described by the message happened. Before using time to group or compare messages, decide which moment matters. section 3.3 section 12.5.1
Table 12.3 Streaming times
| Time |
Responsibility |
Meaning |
| Event time |
Our code |
The time at which an event occurred |
| Create time |
Producer |
The time at which the message was created by the producer |
| Log append time |
Broker |
The time at which the message was written to the log by the broker |
| Stream time |
Stream processor |
The time at which a message is processed |
Compare four useful clocks: event time is when something happened; create time is when the producer created the message; log append time is when the broker wrote it; stream time is when a processor handles it. section 12.5.1
By default, Kafka uses CreateTime: the producer sets the message timestamp when send() is called. That can be later than the event itself. Rather than manually changing this technical timestamp, put event time in a field of the message data. Broker or topic configuration can instead select LogAppendTime, which records when the broker receives the message. section 12.5.1
A customer update from a database
Suppose a customer row changes but keeps the same ID. A JDBC source connector using mode=incrementing and incrementing.column.name=ID will not publish that update: it looks for larger IDs. If the table has a lastupdated column, mode=timestamp with timestamp.column.name=lastupdated can pick up the change. This database column is a way to detect changes; do not confuse it with the Kafka message timestamp. Timestamp-based polling also needs care: changes sharing a timestamp can be missed. section 11.8.3
Partition order follows arrival at the leader broker, not necessarily ascending message timestamps. This matters for time windows that aggregate recent data or correlate topics: messages can arrive too late for the intended window without suitable configuration. Choose the time being used before interpreting a window’s result. section 12.5.1 section 12.7
