Suppose a balance transfer produces one message to debit an account and another to credit an account. Writing only one would leave the result inconsistent. A Kafka transaction groups writes across partitions so they succeed together or are treated as aborted. In a read–process–write pipeline, the transaction can also include the consumed offset, keeping the output and processing position together. section 5.2.2 section 5.2.1
Transactions build on an idempotent producer, which makes writes to individual partitions reliable. Configure a unique `transactional.id`, call `init_transactions()`, begin a transaction, produce the messages, and commit it. Do not run two producers with the same transactional ID at once: a newer producer causes Kafka to fence the older one. section 5.2.2
from confluent_kafka import Producer #1
producer = Producer({
'bootstrap.servers': 'localhost:9092', #2
'acks': 'all', #2
'enable.idempotence': True, #2
'partitioner': 'murmur2_random', #3
'transactional.id': 'transaction-1', #4
})
producer.init_transactions() #5
producer.begin_transaction() #6
producer.produce("customer.balance",
key="bob", value="-10") #7
producer.produce("customer.balance",
key="alice", value="+10") #7
producer.commit_transaction() #8
The Python example writes a debit and credit to `customer.balance` in one transaction. Its `acks: all` and idempotence settings support reliable production; `commit_transaction()` completes the grouped write. section 5.2.2
Set consumers’ `isolation.level` to `read_committed`. Otherwise, they can read uncommitted writes, defeating the transaction’s purpose. Aborting does not delete produced records; Kafka marks them so read-committed consumers ignore them. A transaction coordinator tracks participating partitions and writes commit or abort markers, including when failures interrupt a producer or broker. section 5.2.2 section 5.2.3
Figure 5.7 Our consumer stops receiving new messages as soon as a transaction remains unfinished because the broker only sends messages up to the start of the unfinished transaction and holds back all subsequent messages, even those unrelated to the transaction. The ordering guarantee still holds. When a transaction-commit is received, all messages are released and sent to our consumer. If a transaction is aborted, our consumer doesn’t receive messages belonging to the aborted transaction, but messages not belonging to the transaction are still received normally.
An unfinished transaction can delay a read-committed consumer: Kafka holds back later messages, even unrelated ones, until the transaction commits or aborts. This preserves ordering but can affect how promptly messages arrive. section 5.2.3
For a Kafka-to-database pipeline, use a different boundary: make database writes idempotent, or store the data and consumed offset in one database transaction. A Kafka transaction addresses atomic Kafka writes; it does not replace planning for the database write. section 5.2.1
