Suppose several consumers process an online store’s price updates. Each needs to know how far it has read. Kafka records that progress in __consumer_offsets, an automatically created internal topic. An offset acts like a bookmark: it identifies a position in a partition’s log so a consumer can track its progress without relying only on memory. section 3.1.1 section 4.1.2
$ kafka-topics.sh \
--list \
--bootstrap-server localhost:9092
__consumer_offsets
products.prices.changelog
Listing topics with kafka-topics.sh shows __consumer_offsets alongside the application’s price-change topic. Its double-underscore prefix marks it as an internal Kafka topic. section 3.1.1
Kafka associates stored offsets with consumer groups rather than individual consumer instances. That lets multiple instances in one group share responsibility for partitions. __consumer_offsets is compacted: Kafka regularly cleans up older offsets for each group to save space. section 9.3.1 section 9.5
Starting at the beginning
Run `kafka-console-consumer.sh --topic products.prices.changelog --from-beginning --bootstrap-server localhost:9092` to read existing price messages. Without `--from-beginning`, the console consumer starts at the end and reads only new messages. This flag chooses a starting position for the example; __consumer_offsets is the internal topic used to store consumer-group progress. section 3.1.1 section 9.3.1
Do not create application topics with a double-underscore prefix: Kafka reserves that naming pattern for internal and special-purpose topics. If processing data and recording its offset must succeed together in an external database, storing both in one database transaction is another option; Kafka Connect connectors can use that approach for exactly-once guarantees. section 3.1.1 section 9.3.1
