Idempotency means that repeating an operation produces the same effective result as performing it once. In event-driven architecture, it limits the effects of retries and duplicate events, while shifting consistency concerns toward eventual consistency, idempotency, and out-of-order event handling. section 1.1 section 10.1.3
Kafka producer idempotence prevents duplicate and out-of-order messages within a partition. With enable.idempotence=true, Kafka assigns the producer an identifier and gives each batch a monotonically increasing sequence number for each partition. The broker accepts the expected sequence and rejects out-of-sequence batches with a retriable exception. section 7.4.1
Lost acknowledgment during retries
A producer sends batch 1, loses the acknowledgment, and then sends batch 2 successfully. Without idempotence, resending batch 1 can duplicate data or place it after batch 2. Idempotent production detects the sequence mismatch and enables a correctly ordered retry. section 7.4.1
A Kafka producer uses supporting settings to preserve reliable partition writes:
- Set max.in.flight.requests.per.connection to 5 or fewer. section 7.4.1
- Set retries to a positive number so transient failures can be retried. section 7.4.1
- Set acks to all so acknowledgment follows replication across in-sync replicas. section 7.4.1
Producer idempotence does not make every consumer workflow safe automatically. Topic requirements must ask whether events are idempotent, because duplicate processing can otherwise corrupt data, create inconsistent state, or repeat unintended side effects. section 10.1.3
