How deeply does Apache Kafka in Action cover change-data-capture?

Defines change monitoring, contrasts it with JDBC polling, and walks through PostgreSQL preparation; leaves Debezium installation to its documentation.

Change-data-capture

As explained in Apache Kafka in Action

Change data capture is a technique that monitors and records changes made to a data source rather than repeatedly reading its complete current contents.

A core system may need to feed new applications without being repeatedly queried or modified for each one. Change data capture (CDC) monitors changes to its data rather than repeatedly reading the complete current contents. For a relational database, a connector can follow the transaction log, where changes are recorded, and send the resulting data to Kafka. section 11.9 section 18.2

Why not just poll a table? A JDBC Source Connector can track an incrementing column or timestamp, but those choices can miss changes. An update to an existing row does not give it a new ID; two changes sharing a timestamp can also cause one to be skipped. JDBC bulk mode reloads the entire table and is inefficient. A CDC connector such as Debezium addresses the change-tracking problem instead. section 11.8.3 section 11.10

For a PostgreSQL-to-Kafka setup:

Set PostgreSQL’s wal_level to logical and restart the server so the connector can monitor the necessary changes. section 11.9.1
Create a database user with REPLICATION privilege, then install and configure the Debezium connector. section 11.9.1
Configure database connection details and table filtering; use topic.prefix to organize generated Kafka topic names and avoid conflicts. section 11.10

# Linux file path: /etc/postgresql/<version>/main/postgresql.conf
# Windows file path: C://Programs/PostgreSQL/<version>/data/postgresql.conf
wal_level = logical

The PostgreSQL configuration line needed before the Debezium connector can monitor changes. section 11.9.1

Getting changes into Kafka is not the same as making them ready for other teams. Core-system data may retain unfamiliar formats and be spread across many normalized tables. Build new, usable data structures from the captured data rather than asking downstream applications to consume the raw tables directly; consolidation can require substantial joins and state stores. section 18.2 section 18.6

How Apache Kafka in Action covers it

liveBook manning.com

Read Apache Kafka in Action on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

ScyllaDB in ActionChange data capture is a database feature that records table mutations in a change-log table so they can be consumed over time.
liveBook manning.com

Kafka for ArchitectsChange data capture is a pattern that observes a source system's change records and emits insertions, updates, and deletions as events.
liveBook manning.com

Architecting an Apache Iceberg LakehouseChange data capture processing handles records representing inserts, updates, and deletes made to a source system.
liveBook manning.com

Kafka Streams in Action, Second EditionChange data capture is a technique for detecting and exporting database record changes such as inserts, updates, and deletes as they occur.

Compare how 5 books explain change-data-capture →

Related concepts

transaction loguses · A transaction log is an ordered record of transactions, transaction-related events, or resulting data changes, maintained as a history and often used for tracking or recovery.
CDC tableproduces · A CDC table is a ScyllaDB table that stores records describing changes made to rows in a corresponding source table.
database tableuses · A database table is a relational database structure that organizes records into rows and defined columns.
Apache Kafkaused by · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.
small fileproduces · A small file is a data file whose size is sufficiently low that file-management, metadata, and scheduling overhead become significant relative to its data content.
Debeziumused by · Debezium is an open-source change data capture platform that captures database changes and streams them to downstream systems.

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.