11 Integrating external systems with Kafka Connect
This chapter covers
- Integrating Kafka with data sources and sinks
- Configuring connectors and workers for optimal data flow
- Exploring the REST API for managing Kafka Connect
- Creating and modifying connectors
- Using Java Database Connectivity Source and Debezium connectors
In most cases, we don’t introduce Kafka independently of other systems in our company, but rather we want to connect systems such as databases and messaging systems to Kafka. Most of these use cases are quite similar. We want to transfer data from predefined database tables to specific topics or write data from certain topics into a file. Of course, we always have the option to manually write our own producers and consumers to move data to or from Kafka.
However, this is very time-consuming and often leads to error-prone systems that are also difficult to scale. Even with simple requirements, such as transferring data from one system to another, there are many special cases to consider. An alternative is to use Kafka Connect.