Asynchronous processing hands consumed records to another thread or deferred processor, allowing the Kafka consumer's poll loop to continue without waiting for each record to finish. section 4.3.5
A Kafka consumer must coordinate asynchronous work with offset commits:
- Disable auto-commit when processing may outlast the poll loop; otherwise the consumer can commit offsets before asynchronous work completes. section 4.3.5
- Process records by topic-partition, track the last completed offset, add one, and queue the resulting offsets for commitment. section 4.3.5
- Commit only offsets representing records considered fully processed, avoiding both skipped work after failure and unnecessary reprocessing. section 4.3.5
Asynchronous record processor
After poll() returns a batch, process each partition's records asynchronously, record its final offset plus one, place the offsets in a queue, and have the consumer commit them only after processing completes. section 4.3.5
Figure 4.20 Asynchronous processing with auto-committing can lead to potentially lost records.
Use the figure to trace how auto-committing can advance past records still being processed asynchronously. section 4.3.5
