When one service needs data held by another, writing a producer to publish changes and a consumer to copy them can turn a simple replication need into recurring application code. Kafka Connect offers a configuration-driven alternative. It runs separately from Kafka and moves data between Kafka topics and external systems such as databases, cloud storage, and business applications. section 1.5.2 section 7.3
A pipeline combines an installed connector plugin with a configured connector instance:
- A source connector retrieves data from an external system and publishes it to Kafka. A sink connector reads Kafka topics and writes to an external system. The two operate independently. section 7.3 section 5.1.2
- Configuration specifies the connection and data mapping, such as database tables and Kafka topics. A connector configuration can be deployed through the REST API; installed plugins supply the integration with each technology. section 7.3 section 7.3.1
Replicating operational data
A source connector can load address changes from a customer database into Kafka, while a sink connector delivers them to an order database—without custom producer and consumer code. For a table of immutable transactions, a JDBC source connector can instead use an incrementing ID column to identify new records. Source replication may read current state as snapshots or, with a tool such as Debezium, capture changes from database logs. section 1.5.2 section 7.3.9 section 5.1.2
Figure 5.3 Setting up Kafka Connect for data replication involves using PostgreSQL and MongoDB as source systems. Source connectors are responsible for pulling data from these systems and inserting it into Kafka topics. In turn, sink connectors pull the data from Kafka and insert it into the target systems. In this setup, both sink connectors consume data from the same topics, with one inserting the data into an MS SQL database and the other into Amazon S3.
The diagram shows why sources and sinks need not form one-to-one pairs: PostgreSQL and MongoDB feed Kafka topics, while separate sinks deliver topic data to an MS SQL database and Amazon S3. section 5.1.2
Connectors act as Kafka clients: sources produce, while sinks consume and track partition assignments and offsets through consumer groups. Kafka Connect stores connector configurations, statuses, and source offsets in internal Kafka topics. Connectors may also use Schema Registry when serializing or deserializing messages. These details matter when operating a pipeline, not just creating one. section 7.3.1
Configuration does not eliminate integration design. Basic stateless changes such as renaming fields fit Kafka Connect; logic that depends on prior events or external context calls for a processing layer instead. Check type mappings, whether the source can track changes, and whether loading can be parallelized: gaps can require custom handling, full dumps, or a single slow task. Direct database integration may also expose the database model beyond intended data contracts, so consider isolation and governance alongside deployment speed. section 1.5.3 section 7.3.10 section 7.3.9
