A distributed transaction coordinates operations across independent systems or databases so they either all succeed or all fail. The goal is consistency across system boundaries, but coordination becomes difficult when Kafka participates alongside a database. section 7.4.3 section 3.2.6
Cross-system workflows expose two related failure dilemmas:
- Updating the database before publishing can leave systems inconsistent when Kafka publication fails. section 3.2.6 section 7.4.3
- Publishing before updating the database can emit an event for a state change that never commits. section 3.2.6 section 7.4.3
- Consumers face a similar choice: committing Kafka offsets first risks data loss, while storing database results first risks duplicate processing. section 4.3.7
Two-phase commit (2PC) addresses this class of problem through prepare and commit phases. Participants first report whether they are ready; the transaction manager commits all participants only when every participant is ready, otherwise it aborts the transaction. Kafka currently does not support 2PC, although KIP-939 proposes participation in the protocol. section 7.4.3
Profile update and notification
When a user changes a profile, ProfileService must save the new profile in its transactional database and notify other systems through Kafka. Because those actions cannot form one Kafka-supported 2PC transaction, the transactional outbox pattern uses a separate process to publish events to Kafka. section 7.4.3
Figure 7.28 When the transactional outbox pattern is used, a separate process (step 4 in the figure) publishes events to Kafka.
Use the architecture figure to trace how a separate publishing process supports the transactional outbox pattern. section 7.4.3
