A producer may write a message successfully but never receive its acknowledgment. If it retries, an ordinary write can put the message in the log twice. An idempotent write makes that retry harmless: repeating the operation does not add another copy. This matters when reliable delivery requires retries but duplicate records would be a problem. section 5.1.3 section 5.0
Figure 5.4 If idempotency is enabled, the producer sends its producer ID with each message and also a sequence ID for each message. This allows the broker to detect if messages are missing, appear in the wrong sequence, or have been sent multiple times, and then to respond accordingly.
The producer ID and message sequence ID let the partition leader recognize a repeated message or a gap in the sequence. For a message already persisted, the leader resends the acknowledgment rather than writing it again; for an out-of-order message, it returns a negative acknowledgment so the producer can retry. section 5.1.3
Enable idempotent production
Run `kafka-console-producer.sh --topic products.prices.changelog.min-isr-2 --bootstrap-server localhost:9092 --producer-property acks=all --producer-property enable.idempotence=true`. Idempotence also requires retries greater than zero and `max.in.flight.requests.per.connection` no greater than five. A delivery timeout bounds retrying. section 5.1.3
Producer idempotency addresses repeated writes to a Kafka partition; it does not by itself make a Kafka-to-database write and its consumer-offset commit atomic. If the database write succeeds but the offset is not committed, the consumer can process the message again. One approach is to make the database write idempotent, then commit the offset. Another is to store the offset with the data in a single database transaction. section 5.2.1 section 5.4
Make a database insert safe to repeat
For a price-changelog row in PostgreSQL, use `INSERT INTO products_price_changelog (id, timestamp, name, price) VALUES (…) ON CONFLICT DO NOTHING;`. When the row already exists, the repeated insert makes no additional change. section 5.2.1
