liveBook · Manning

Transaction log miner

As explained in Microservices Patterns, Second Edition

A software component that reads database transaction-log entries and converts relevant changes into messages.

A transaction log miner reads committed database-log entries, turns relevant changes into messages, and publishes them to a message broker. It can relay messages from an RDBMS outbox table or messages appended to NoSQL records. section 8.3.8

This approach can avoid the querying limitations or performance drawbacks of polling an outbox table or business entities. Its fit depends on the database’s available log or change-stream capabilities. section 8.3.8

Examples of implementations and data sources include:

  • Debezium publishes database changes to Apache Kafka. section 8.3.8
  • DynamoDB Streams provides a time-ordered sequence of table-item changes that an application can read and publish as events. section 8.3.8
  • Eventuate CDC reads outbox changes using MySQL binlog, Postgres WAL, or polling, and publishes them to Apache Kafka. section 8.3.8
Figure 8.18 A service publishes messages inserted into the OUTBOX table by mining the database’s transaction log.
TransactionLogMining

Diagram of a service publishing outbox messages by mining the database transaction log. section 8.3.8

How Microservices Patterns, Second Edition covers it

How deep it goes
Defines the miner’s read-convert-publish flow and supported message sources, but does not detail its internals.
Examples
Cites Debezium, DynamoDB Streams, and Eventuate CDC as practical change-capture examples.
What you can do after
Apply it via Debezium or DynamoDB Streams to publish database changes as messages or events.
How it connects
Connects transaction outbox, transaction log tailing, database change streams, and message brokers.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Microservices Patterns, Second Edition. The section links point to the parts of the book it draws on.