A sink connector moves records from Kafka topics into an external system. For example, address changes published to Kafka can be written to an order service’s database without a custom consumer application. A JDBC sink connector uses a database integration plugin and an appropriate JDBC driver to write to a relational database. section 1.5.2 section 7.3.8
Kafka Connect runs the connector from a configuration that identifies the integration and its topic-to-table mappings. The connector acts as a Kafka consumer: its tasks share a consumer group, divide assigned partitions, and use offsets to track progress. Source and sink connectors operate independently, so one set of topics can feed different destinations. section 7.3 section 7.3.1 section 7.3.6 section 5.1.2
A converter deserializes each Kafka message into a ConnectRecord; the sink connector then translates that record into the target system’s format. The converter is selected for the format stored in Kafka, rather than for the destination technology. section 7.3.3
Replicating reference data
Source connectors can publish reference-table data from PostgreSQL and MongoDB to Kafka. Two independent sink connectors can consume the same topics, writing to an MS SQL database and Amazon S3 respectively. This keeps the destinations separate while giving them a shared integration path. section 5.1.2
Plan for records the destination cannot accept. A database integrity-constraint violation, for instance, will not be fixed by retrying the same insert. The connector can fail and require intervention, or route the problematic message to a dead-letter topic while processing continues. Either choice needs monitoring; the dead-letter topic also needs alerts. section 7.3.6
A sink connector is useful when a configured Kafka pipeline is preferable to writing consumer code. It is not the only replication choice: database synchronization tools may provide stronger consistency guarantees, while a Kafka-based path allows data in topics to be enriched or transformed before delivery. section 1.5.2 section 5.1.2
