A transaction log is an ordered record of database changes. Recording changes before applying them supports durability and traceability. In Kafka, the related commit-log pattern appends immutable messages sequentially, allowing consumers to replay them and reconstruct state. section 1.4.3
Key architectural properties include:
- Preserves sequence within a log or partition, while Kafka does not provide global ordering across a topic. section 1.4.3 section 2.2.1
- Uses immutable records, retention settings, and offsets for retrieval and processing progress rather than individual message edits or deletions. section 1.4.3 section 2.2.1
- Supports parallel processing by dividing a topic into partitions, each with its own ordered sequence and offset range. section 2.2.1
Database change capture
A connector can read a database transaction log directly instead of issuing SELECT queries. Debezium-style capture can expose inserts, updates, and deletes without schema modifications, but exposing the log creates security concerns. JDBC sinks provide an alternative with insert or idempotent upsert behavior; tombstones can delete matching database records. section 7.3.8
Figure 2.4 A topic is divided into partitions, each with its own index. When a producer sends a message, it specifies the partition to which the message should be directed.
Shows how a topic divides into indexed partitions, with producers directing messages to partitions and each partition maintaining its own commit-log sequence. section 2.2.1
