5 Kafka Connect
This chapter covers
- Getting started with Kafka Connect
- Applying Single Message Transforms
- Building and deploying your own connector
- Making your connector dynamic with a monitoring thread
- Creating a custom transformation
This chapter will teach you how to move events into and out of Apache Kafka quickly. While Kafka can function as a central nervous system for data, it primarily provides a decoupled and centralized approach to data access. Still, other essential services, like full-text searching, report generation, and data analysis, can only be serviced by applications specific to those purposes. No single technology or application can satisfy all the needs of a business or organization.
We’ve established in earlier chapters that Kafka simplifies the architecture for a technology company by ingesting events once, and any group within the organization can consume events independently. In the previously mentioned cases where the consumer is another application, you’ll need to write a consumer specifically for that application. You’ll repeat a lot of code if you have more than one. A better approach would be to have an established framework that you can deploy that handles either getting data from an application into Kafka or getting data out of Kafka into an external application. That framework exists and is a critical component of Apache Kafka: Kafka Connect.