A producer can write a message successfully yet never receive its ACK. If it retries, the log may contain a duplicate. For a sensor reading that may be tolerable; for a banking transaction, it could mean crediting or debiting an amount twice. Exactly-once delivery aims to preserve one durable effect despite that uncertainty. section 5.1.3
The delivery guarantee depends on what failures the configuration permits:
- At most once avoids duplicates but may lose messages, as with `acks=0`. section 5.1.3
- At least once uses `acks=all` and sufficient in-sync replicas to protect delivery, but retries can create duplicates. section 5.1.3
- Exactly once adds an idempotent producer to `acks=all`. Producer and sequence IDs let the broker recognize repeated or out-of-order messages. section 5.1.3
Enable idempotent production
For a Kafka console producer, set `--producer-property acks=all` and `--producer-property enable.idempotence=true`. This addresses duplicate production; a workflow that consumes, processes, and writes data must also coordinate its output with its consumed offset. section 5.1.3 section 5.2.2
For Kafka-to-database writes, one option is to write first, then commit the Kafka offset, making repeated database writes idempotent—for example, with an `UPSERT`. Another is to store the offset alongside the data in one database transaction, so both changes commit atomically. section 5.2.1
For Kafka-to-Kafka processing, use a producer transaction to write output and commit consumed offsets together. Disable the consumer’s automatic offset commits and send offsets through the transactional producer; otherwise, the offset can advance separately and cause loss or duplication. Give the producer a unique `transactional.id`, and configure consumers with `read_committed` so they do not read uncommitted output. Transactions add performance overhead. section 5.2.2 section 5.4
from confluent_kafka import Producer #1
producer = Producer({
'bootstrap.servers': 'localhost:9092', #2
'acks': 'all', #2
'enable.idempotence': True, #2
'partitioner': 'murmur2_random', #3
'transactional.id': 'transaction-1', #4
})
producer.init_transactions() #5
producer.begin_transaction() #6
producer.produce("customer.balance",
key="bob", value="-10") #7
producer.produce("customer.balance",
key="alice", value="+10") #7
producer.commit_transaction() #8
The Python producer example shows transaction setup, two balance updates, and a single commit. It illustrates atomic production; a consume-process-produce application must also include its consumed offsets in that transaction. section 5.2.2
