When applications need database changes without repeatedly querying a core system, Debezium can capture those changes and send them to Kafka. It uses change data capture (CDC): instead of reading only a table’s current contents, a connector monitors changes recorded by the database. For PostgreSQL, the connector reads from the database’s transaction log. section 11.9 section 18.2
To run the PostgreSQL example:
- Set wal_level = logical in postgresql.conf and restart PostgreSQL; otherwise, the required changes cannot be monitored. section 11.9.1
- Create a database, a customers table, and a database user with the REPLICATION privilege. Install the Debezium connector before creating its Kafka Connect configuration. section 11.9.1
- Configure the PostgreSQL connector with database connection settings, plugin.name set to pgoutput, publication.autocreate.mode set to filtered, topic.prefix set to debezium, and table.include.list set to public.customers. section 11.9.2
{
"name": "customers_debezium_connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "localhost",
"database.port": "5432",
"database.user": "customers_user",
"database.password": "supersecret",
"database.dbname" : "customers",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": false,
"plugin.name": "pgoutput",
"publication.autocreate.mode": "filtered",
"topic.prefix": "debezium",
"table.include.list": "public.customers"
}
}
The JSON configuration brings together the Kafka Connect class, PostgreSQL connection, extraction settings, topic prefix, and table filter. section 11.9.2
Watch a row change
Submit the configuration to the Kafka Connect API with curl, then consume debezium.public.customers from the beginning. The topic name combines topic.prefix with the included table name. Existing rows appear first with op: r, indicating an initial-snapshot record. Insert a row to see op: c; update one to see op: u. Each message carries the changed row in after, origin details in source, and an operation code in op. A before value may be available if the database and connector are configured to include it. section 11.9.3
The full change envelope is useful for understanding what happened, but consumers may need only the current row. Kafka Connect’s ExtractNewRecordState single-message transform can write just after to Kafka. For broader reuse, avoid exposing raw core-system tables as an application-facing format: their data may be normalized across many tables and difficult for other teams to interpret. Build suitable downstream data structures instead; consolidating many tables can itself require complex joins and state stores. section 11.9.3 section 18.2
