liveBook · Manning

Transaction log tailing

As explained in Microservices Patterns, Second Edition

A messaging pattern that publishes messages derived from changes read from a database transaction log.

A transaction log miner reads committed database log entries and publishes corresponding changes as messages to a broker. In a transaction outbox design, it converts log entries for inserted outbox messages into broker messages. section 8.3.8 section 8.4

Consider log tailing when polling the database is too costly or unsuitable:

  • Polling is simple and can work at low scale, but frequent queries can be expensive; querying business entities may also be inefficient or unsupported in some NoSQL databases. section 8.3.8
  • Log tailing is a more sophisticated and performant alternative for publishing changes from an RDBMS outbox or messages appended to NoSQL records. section 8.3.8

Implementations

Debezium publishes database changes to Apache Kafka. DynamoDB Streams exposes a time-ordered sequence of table-item changes that an application can read and publish as events. section 8.3.8

Figure 8.18 A service publishes messages inserted into the OUTBOX table by mining the database’s transaction log.
TransactionLogMining

Diagram of a service publishing outbox messages by mining the database transaction log. section 8.3.8

How Microservices Patterns, Second Edition covers it

How deep it goes
Traces log entries through a miner to broker publication, without detailing miner configuration or operation.
Examples
Names Debezium with Apache Kafka and DynamoDB Streams as implementations for publishing database changes.
What you can do after
Evaluate log tailing as an alternative when frequent database polling is costly or impractical.
How it connects
Connects transaction outbox, polling publisher, and message broker as parts of message publication.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Microservices Patterns, Second Edition. The section links point to the parts of the book it draws on.