A Kafka Connect worker is a single process in a Kafka Connect cluster. It manages connectors and runs their tasks, such as reading from source partitions or writing to destinations. section 11.2.1
In Distributed Mode, workers with the same group.id coordinate through Kafka’s rebalancing protocol. Tasks can move between workers when a worker fails or when capacity is added, supporting parallel processing and fault tolerance. section 11.3
A worker participates in several operational responsibilities:
- Loads connector plugins from plugin.path and starts with worker.properties. section 11.2.1
- Uses internal Kafka topics for connector configuration, offsets, and status; production deployments should generally use replication factor three. section 11.2.1
- Shares connector tasks across workers; for example, four tasks importing twelve database tables can handle three tables per task. section 11.3
- Exposes a JSON REST API on port 8083 for plugin discovery, connector inspection, and task and worker status. section 11.2.1 section 11.5.1
Scaling a database import
Start several workers in Distributed Mode with the same group.id, configure a connector with tasks.max set to 4, and let Kafka Connect distribute twelve table-import tasks across the workers. Adding a worker causes tasks to be redistributed; losing one causes the remaining workers to take over its tasks. section 11.3
bootstrap.servers=localhost:9092 group.id=connect config.storage.topic=connect-config offset.storage.topic=connect-offset status.storage.topic=connect-status key.converter=org.apache.kafka.connect.storage.StringConverter value.converter=org.apache.kafka.connect.storage.StringConverter plugin.path=<PATH-TO-YOUR-USER-DIRECTORY>/kafka/libs/
A minimal worker.properties listing shows the cluster identity, internal topics, converters, and plugin location that a worker needs. section 11.2.1
$ curl http://localhost:8083/connectors/customers-source/status | jq
{
"name": "customers-source",
"connector": {
"state": "RUNNING",
"worker_id": "127.0.0.1:8083"
},
"tasks": [
{
"id": 0,
"state": "RUNNING",
"worker_id": "127.0.0.1:8083"
}
],
"type": "source"
}
A connector status response identifies the worker hosting the connector and each task, together with running, paused, or failed states. section 11.5.1
