Pipelining overlaps Kafka record retrieval and processing: the consumer hands one batch to asynchronous work, then returns to poll for the next batch. section 4.3.5
A reliable pipeline separates three responsibilities:
- Consume a batch and hand it to a record processor that returns immediately. section 4.3.5
- Process records concurrently while tracking which records have fully completed. section 4.3.5
- Commit only completed offsets, using the next offset after the last processed record in each partition. section 4.3.5
Consumer loop
Poll records, call recordProcessor.processRecords(consumerRecords), obtain recordProcessor.getOffsets(), and call consumer.commitSync(offsetsAndMetadata) only when completed offsets are available. The processor groups records by TopicPartition, processes each record, stores lastOffset + 1, and offers the offset map for commitment.
