Automatic offset commit lets KafkaConsumer periodically record the offsets for consumed records without an explicit commit call. Enable it with enable.auto.commit=true; auto.commit.interval.ms controls the elapsed time between commit checks. section 4.3.5
A poll retrieves a batch, the application processes it, and the consumer checks the interval before starting the next poll. When the interval has passed, Kafka commits the previous batch’s highest offset plus one, marking those records as consumed. section 4.3.5
Figure 4.17 With automatic commits enabled when returning to the top of the poll loop, the consumer commits the highest offset +1 of the previous batch if the auto-commit interval has passed.
Trace the poll-loop timing and the highest-offset-plus-one value used by automatic commits. section 4.3.5
Keep these operational consequences in mind:
- Kafka stores committed offsets in the internal __consumer_offsets topic. section 4.3.5
- Failures before a commit can cause records to be processed again after restart. section 4.3.5
- Asynchronous processing combined with auto-commit can potentially mark records before processing finishes, losing them on failure. section 4.3.5
