liveBook · Manning

Outbox pattern

As explained in Microservices Patterns, Second Edition

A messaging pattern that atomically records an outgoing message alongside a business-data change in the same database transaction, then publishes the message afterward.

When a service changes business data and sends a message, a failure between the two operations can leave the system inconsistent. A distributed transaction spanning the database and broker is one alternative, but many modern brokers do not support it. section 8.3.8

The outbox pattern records the outgoing message alongside the business-data change, then publishes it separately. With a relational database, the service inserts the message into an OUTBOX table in the same local ACID transaction that creates, updates, or deletes the business object. A message relay reads the table and sends messages to the broker. The atomic operation is the database update and message record, not the later broker publication. section 8.3.8 section 8.4

Figure 8.17 A service reliably publishes a message by inserting it into an OUTBOX table as part of the transaction that updates the database. The Message Relay reads the OUTBOX table and publishes the messages to a message broker.
ReliablePublication

The diagram separates the local database transaction from the relay that publishes to the broker. section 8.3.8

Choose how the relay retrieves pending messages:

  • Polling publisher: retrieve messages from the database by polling. section 8.4
  • Transaction log tailing: retrieve messages by mining the database transaction log. section 8.3.8 section 8.4

A similar approach can work with some NoSQL databases: append a pending message to a list on the business entity in the same database operation that updates it. The challenge is efficiently finding entities with pending messages. section 8.3.8

Eventuate messaging

An Eventuate service calls MessageProducer.send() rather than publishing directly to the broker. The call uses the transaction outbox; a separate Eventuate CDC service publishes messages from the OUTBOX table. On the receiving side, an idempotent consumer detects and discards duplicates. section 8.3.10

Use an outbox when a business update must reliably lead to a message without a distributed database-and-broker transaction. Plan for the separate relay and for duplicate delivery: a consumer that is not naturally idempotent can track processed messages and discard duplicates. section 8.3.8 section 8.4

How Microservices Patterns, Second Edition covers it

How deep it goes
Explains relational and NoSQL recording and identifies polling and transaction-log relay options, without detailing their procedures.
Examples
Illustrates message publication through Eventuate MessageProducer and a separate CDC service.
What you can do after
Apply it via the Eventuate platform using its messaging API and separate CDC service.
How it connects
Connects local ACID transactions, message relays, polling publishers, transaction log tailing, and idempotent consumers.

Also explained in

Compare how 4 books explain outbox pattern →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Microservices Patterns, Second Edition. The section links point to the parts of the book it draws on.