liveBook · Manning

Kafka Connect cluster

As explained in Apache Kafka in Action

A Kafka Connect cluster is a group of Kafka Connect instances with the same group identifier that coordinate and distribute connector load.

A Kafka Connect cluster is one or more workers running in Distributed Mode with the same group.id. The workers coordinate through Kafka topics and distribute connector tasks among themselves, allowing external systems to exchange data with Kafka without placing all work on one process. section 11.2 section 11.3

A cluster provides several operational benefits:

  • Adding workers with the same group.id increases available capacity; Kafka Connect redistributes tasks across them. section 11.3
  • If a worker crashes, the remaining workers redistribute its tasks. section 11.3
  • Multiple connectors can run simultaneously, such as importing database data while writing to a third-party system. section 11.3
  • Tasks run independently and can process partitions or other units of connector work in parallel. section 11.2.1
bootstrap.servers=localhost:9092
group.id=connect
config.storage.topic=connect-config
offset.storage.topic=connect-offset
status.storage.topic=connect-status
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter   
plugin.path=<PATH-TO-YOUR-USER-DIRECTORY>/kafka/libs/

A Distributed Mode worker configuration identifies the Kafka cluster, worker group, internal configuration topics, offset topics, status topics, converters, and plugin path. section 11.2.1

Scaling database imports

Suppose a connector imports 12 database tables and tasks.max is 4. Kafka Connect creates four independent tasks, each responsible for three tables, then distributes those tasks across workers. Adding another worker with the same group.id lets the cluster rebalance the workload. For production deployments, use connectors whose source data is available to every worker; the local-disk FileStreamSource connector is unsuitable for production because files may not exist on other workers. section 11.3

$ connect-distributed.sh worker.properties

Start a configured Distributed Mode worker with connect-distributed.sh and worker.properties; Kafka Connect exposes configuration and monitoring through its REST API, normally on port 8083. section 11.2.1 section 11.5.1

How Apache Kafka in Action covers it

How deep it goes
Explains worker coordination, internal topics, task distribution, scaling, failures, configuration, and REST inspection; defers detailed parameter explanations.
Examples
Uses a 12-table database import, tasks.max set to 4, FileStreamSource limitations, worker.properties, and curl status queries.
What it assumes
Assumes Kafka Connect workers and connectors exist, while defining workers, tasks, Distributed Mode, and internal Kafka topics.
What you can do after
Operate/configure a Distributed Mode cluster, scale it by adding workers, and inspect connectors, tasks, plugins, and failures through REST endpoints.
How it connects
Connects Distributed Mode with Kafka topics, consumer groups, Kafka Rebalance Protocol, connectors, tasks, and REST API operations.
Cover of Apache Kafka in Action
liveBookmanning.com
Read Apache Kafka in Action on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.