At-least-once delivery guarantees that records are not lost, while allowing a record to be delivered or processed more than once. Kafka provides this guarantee by default. section 4.2.2 section 4.7
Duplicates arise when successful processing is not followed by a durable acknowledgment:
- A producer may resend a batch when the broker appends it but the acknowledgment does not arrive in time. section 4.2.2
- A consumer may process records and fail before committing offsets, then restart from the last committed offset and process them again. section 4.3.5
- Processing records synchronously before committing helps preserve the guarantee; asynchronous handoff can commit offsets before processing finishes and lose records. section 4.3.5
Commit after processing
A consumer polls a batch, passes it to recordProcessor.processRecords, retrieves the resulting offsets, and calls consumer.commitSync(offsetsAndMetadata). If the application fails before the commit, the batch is read again; processing therefore needs to tolerate duplicates. section 4.3.5
