Connector

As explained in Kafka for Architects

A connector is a configured Kafka Connect instance that defines a particular data pipeline between Kafka and an external system.

A connector is a configured data pipeline between Kafka and an external system. It runs in a Kafka Connect cluster using an installed connector plugin: the plugin supplies the integration capability, while the connector’s configuration defines a particular flow. This can replace custom producer and consumer code when the job is to transfer data rather than design application-specific notifications. section 1.5.2 section 5.1.2

Direction determines the connector’s role:

• A source connector reads an external system and publishes data to Kafka topics. section 7.3
• A sink connector reads Kafka topics and sends data to an external system. Source and sink connectors operate independently. section 5.1.2 section 7.3

Replicating data to multiple destinations

Source connectors can move data from PostgreSQL and MongoDB into Kafka topics. Independent sink connectors can then consume the same topics, one writing to an MS SQL database and another to Amazon S3. For database replication, a connector might read current state through JDBC or stream changes from source logs through a CDC tool such as Debezium. section 5.1.2

Figure 5.3 Setting up Kafka Connect for data replication involves using PostgreSQL and MongoDB as source systems. Source connectors are responsible for pulling data from these systems and inserting it into Kafka topics. In turn, sink connectors pull the data from Kafka and insert it into the target systems. In this setup, both sink connectors consume data from the same topics, with one inserting the data into an MS SQL database and the other into Amazon S3.

Trace the source-to-topic and topic-to-sink flows to see how one set of topics serves separate destinations. section 5.1.2

Configuration specifies details such as database connections, table-to-topic mappings, and polling frequency. Install a plugin suited to each external technology before configuring its connectors. Propagation speed is not fixed: connector settings, external-system performance, network bandwidth, and Kafka Connect capacity all matter. Treat connector settings as project artifacts alongside topic configurations and schemas so the integration remains manageable as the Kafka project grows. section 5.1.2 section 7.3 section 10.2

How Kafka for Architects covers it

How deep it goes

Defines plugins, connector direction, configuration, and replication approaches; stops short of a complete configuration walkthrough.

Examples

Illustrates database-to-Kafka replication, independent MS SQL and Amazon S3 sinks, and JDBC or Debezium source options.

What it assumes

Assumes familiarity with Kafka topics and external data systems while defining the distinction between plugins and configured connectors.

What you can do after

Operate/configure it by selecting an appropriate plugin, defining source or sink mappings, and accounting for propagation constraints.

How it connects

Links Kafka topics, connector plugins, source and sink connectors, snapshot-based replication, and change data capture.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Kafka Streams in Action, Second EditionA connector is a Kafka Connect component that coordinates tasks for moving data between Kafka and an external system.
• 

  liveBookmanning.com
  Azure SecurityA data connector is an integration component that collects data from a source and makes it available to a monitoring or analytics system.
• 

  liveBookmanning.com
  Apache Kafka in ActionA Kafka Connect connector is a component that transfers data between Kafka and an external system according to a configured connection definition.
• 

  liveBookmanning.com
  AI for Everyday ITA connector is a software integration component that exposes access to an external service or data source for an application or workflow.
• 

  liveBookmanning.com
  Quarkus in ActionA connector is a MicroProfile Reactive Messaging service-provider interface implementation that bridges a messaging channel to an external messaging provider.
• 

  liveBookmanning.com
  Data Analysis with LLMsAn integration component that enables a language model to access external data sources such as web search services or databases.

Compare how 7 books explain connector →

Related concepts

• source connectorincludes · A source connector is a Kafka Connect connector that reads data from an external source system and publishes records to Kafka.
• Kafka Connectused by · Kafka Connect is a framework for reliably transferring data between Apache Kafka and external systems through configurable source and sink connectors.
• sink connectorincludes · A sink connector is a Kafka Connect connector that reads records from Kafka and exports them to an external destination system.
• data sourceuses · A data source is an origin, system, repository, or process from which data is obtained for use or processing.
• Azure Logic Appsused by · Azure Logic Apps is Microsoft's managed cloud service for designing and running automated workflows that integrate applications, services, data, and events.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.