4 Building custom connectors with Kafka Connect

 

This chapter covers

  • Developing a custom source connector with Kafka Connect
  • Handling errors in connectors
  • Persisting connector offsets with Kafka Connect
  • Testing connectors with unit tests

The last chapter introduced Kafka Connect and showed how to use community connectors for integrating Kafka topics with external data systems without writing code. In particular, we applied the Debezium source connector for streaming order data from Excellent Toy’s PostgreSQL database to Kafka and used a sink connector for integrating Kafka with Excellent Toy’s data warehouse.

4.1 Using change data capture with HTTP APIs

4.2 Under the hood of custom connectors

4.2.1 Setup

4.2.2 Implementing the custom connector

4.2.3 Packaging and deployment

4.3 Handling errors in connectors

4.4 How to keep track of the state of the data processing

4.5 Testing connectors

4.6 Summary