A Kafka offset is an integer that identifies a record’s position within a partition. Records are appended to that partition’s ordered log, starting at offset 0 and receiving increasing offsets. The combination of topic, partition, and offset uniquely identifies a record; an offset alone does not. Kafka preserves order within a partition, not across a topic’s partitions. section 2.2.1 section 2.4.3
Offsets also let consumers record where to resume. After polling and processing a batch, a consumer can commit its position—the offset of the next record to read. Kafka stores committed positions per consumer group in the internal __consumer_offsets topic. A commit records a place in the stream; it does not prove that processing succeeded. section 4.2 section 4.2.4
A restart after a premature commit
Suppose a consumer commits its position before finishing a database update for a record. If it fails, it resumes after that position and can skip the unfinished work. Automatic commits are convenient but can create this risk; for critical processing, commit manually after the work completes. If processing fails before a commit, the record can be read again. section 4.3.6 section 4.2.4
The position can also be used deliberately:
- Start a consumer at a specified offset, or find an offset by timestamp, to read a chosen range—for example, to populate a microservice’s local store. Explicit positioning is programmed rather than set through consumer configuration. section 4.2.11
- Compare a group’s committed offset with the latest produced offset to estimate consumer lag. Missing offsets, notably after compaction, make this calculation approximate. With workers completing out of order, advance the commit only through the highest contiguous offset whose work has finished. section 4.3.1
- Diagnose a repeatedly failing invalid record by locating its offset. A command-line offset reset can move a stuck consumer past it, while a dead-letter topic provides a route for review. section 6.5.1
Offsets keep increasing when retention removes records; deleted positions are not reused. The word also has a related meaning in Kafka Connect: a source connector can store a source-system position, such as a database updated_time value, in connect-offsets. That value tracks what to load next, rather than identifying a Kafka record’s partition position. section 2.4.3 section 7.3.5
