Producer idempotence protects event delivery when asynchronous requests are retried. With enable.idempotence=true, Kafka prevents duplicate batches and preserves their order within each partition, addressing lost acknowledgments and delayed retries. section 3.2.5 section 7.4.1
Kafka assigns the producer a unique identifier and each batch a monotonically increasing sequence number per partition. The broker accepts the expected sequence, while an out-of-sequence batch produces a retriable error so the producer can resend it correctly. section 7.4.1
Use idempotence with the surrounding delivery settings:
- Set retries to a positive value so transient failures can be recovered. section 7.4.1
- Set acks=all so replicated in-sync brokers acknowledge the batch before delivery is considered successful. section 3.2.5
- Keep max.in.flight.requests.per.connection at 5 or fewer to avoid concurrency issues while maintaining ordering. section 7.4.1
A failed acknowledgment
A producer sends batch 1, loses its acknowledgment, and sends batch 2, which succeeds. Without protection, retrying batch 1 could append it after batch 2. Idempotence lets Kafka detect the sequence mismatch and retry batch 1 in the correct order. section 7.4.1
The setting is useful in enterprise producer configurations such as a Customer 360 operational data store, where all acknowledgments, serializers, partitioning, and producer identifiers must be governed consistently. section 3.2.5
