A sink connector consumes records from Kafka and writes them to an external destination. It provides a standardized path to databases, object storage, messaging systems, and data warehouses while supporting scalable, fault-tolerant data movement. section 11.1
Figure 11.1 Kafka Connect integrates third-party systems with Kafka.
Use the integration diagram to trace Kafka Connect’s two-way relationship with external systems and distinguish sink connectors from source connectors. section 11.1
Configure the connector class to select the destination implementation, such as io.confluent.connect.jdbc.JdbcSinkConnector. Specify topics directly with topics or select them with topic.regex; regular expressions are useful when the topic set changes over time. The tasks.max setting requests workload parallelization, although partitions and connector limits may cap the actual task count. section 11.6.1 section 11.6.1
Production operation commonly includes:
- Managing connector creation, updates, deletion, and status through the Kafka Connect REST API. section 11.10
- Handling failures with retries, error logging, tolerance settings, and a dead-letter queue for unprocessable sink records. section 11.10
- Applying Single Message Transformations to rename, mask, extract, or re-key fields, while reserving complex ETL for other processing. section 11.10
- Using a connector-specific consumer group and considering exactly-once support when the integration requires stronger delivery guarantees. section 11.6.1
