An offset is the broker-assigned ID and logical position of a record within a topic partition. It starts at 0 and increases by 1 as records are appended. Because records are appended at the end of a partition log, offset order gives Kafka’s ordering guarantee within that partition—not across partitions, and not necessarily by event time. section 2.4.1
Offsets let a consumer say where reading should begin. A fetch request names a topic partition and starting offset; the broker returns a batch from that point. Reading does not remove records or change their availability to other consumers. section 2.3
Continue after a processed batch
If a consumer reads offsets 10 through 20, its next read begins at offset 21. To continue after a restart or failure, commit the position only after successful processing. A committed position is the highest successfully consumed offset plus one; with automatic commits, KafkaConsumer commits based on enable.auto.commit and auto.commit.interval.ms. section 4.3.5
Use offsets with their partition-specific meaning in mind:
- Treat an offset as unique within its partition. The same numeric offset in another partition identifies a different record. section 2.4.1
- Do not infer event time from offset order; records are ordered by arrival at the broker. section 2.4.1
- Commit after processing when progress must survive restarts. A restart before a commit can cause records to be processed again. section 4.3.5
- Avoid assuming automatic commits match asynchronous processing completion; committing can potentially mark records consumed before that work finishes. section 4.3.5
An offset is logical, not a byte address. On disk, a broker records each record’s offset, size, and physical segment position. Its memory-mapped index maps offsets to positions so the broker can locate a fetch starting point with a binary search. Index entries store offsets relative to a segment’s base offset, keeping each offset and position entry to 4 bytes. section 2.6.3
Figure 2.5 Offsets indicate where a consumer has left off reading records.
Trace how a consumer that has read one range uses the next offset as its next starting position. section 2.4.1
Record produced by a Spring Kafka application
A produce callback can inspect RecordMetadata after a send completes and log the destination topic, assigned offset, and timestamp. This makes the broker’s assigned position visible to application code while processing a loan application. section 12.2.2
