A consumer reading a long-running stream needs to know where to resume. An offset is a position marker: Kafka brokers assign one when a record is written to a partition, starting at 0 and increasing for subsequent records. It identifies a record’s position within that partition, not its content or a position across the whole topic. section 4.1.2 section 4.5
A consumer uses offsets as bookmarks. It reads forward from a position and tracks where to read next; when no new records are available, that position need not change. Offsets can have gaps if records are later deleted, but remaining records keep their original offsets and Kafka delivers the next available record. section 4.1.3 section 9.3
# Window Consumer 2
$ kafka-console-consumer.sh \
--topic products.prices.changelog \
--from-beginning \
--bootstrap-server localhost:9092
coffee pads 10
coffee pads 11
coffee pads 12
coffee pads 10
The console consumer’s --from-beginning option reads existing records from the earliest available offset. Without it, the command starts at the end and reads only new records. section 3.1.1
Choose where to keep the bookmark according to what the consumer must recover:
- For consumer groups, Kafka stores offsets in its internal __consumer_offsets topic so consumers can track their progress. section 9.5
- An in-memory cache that must be rebuilt after a crash may deliberately reread records rather than store an offset. section 9.3.1
- A service writing to a database can store data and its offset in one transaction, so both are recorded or neither is. section 9.3.1
If a consumer has no known offset for a partition, auto.offset.reset can select the earliest available record or the end. Kafka Connect applies the same position-tracking idea to source streams: a database might use sequential IDs or timestamps, while a file might use a line or byte position. An offset is not a search key; querying records by content belongs in a suitable database or search system. section 9.3.1 section 11.1 section 4.1.3
