Kafka Connect worker

As explained in Kafka Streams in Action, Second Edition

A Connect worker is a Kafka Connect process that runs connector instances and their tasks.

A Kafka Connect worker is a Connect process that runs connector instances and their tasks. It operates separately from the Kafka broker and receives server-level configuration, while each connector supplies its own configuration. section 5.3

Configure the worker to provide shared runtime behavior:

• Set key.converter and value.converter as defaults for connectors; individual connectors can override them. section 5.3
• Set plugin.path so the worker can locate connectors, converters, and their dependencies. section 5.3
• Use standalone mode for local development; use distributed mode when parallelism and fault tolerance matter in production. section 5.3

Figure 5.3 Connect in standalone mode; all tasks reside in one worker.

Compare standalone deployment, where all tasks reside on one worker, with distributed deployment, where tasks spread across connector instances. section 5.3

How Kafka Streams in Action, Second Edition covers it

How deep it goes

Explains worker configuration, converters, plugins, and deployment modes, but stops short of worker internals.

Examples

Uses Docker setup, converter configuration, standalone and distributed diagrams, and a JDBC source connector REST request.

What you can do after

Operate and configure a Kafka Connect worker for local development or distributed production deployment.

How it connects

Connects connector instances, tasks, converters, plugins, Kafka brokers, and distributed fault tolerance.

liveBookmanning.com

Read Kafka Streams in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Apache Kafka in ActionA Kafka Connect worker is a process in a Kafka Connect cluster that manages connectors and runs their tasks.

Compare how 2 books explain Kafka Connect worker →

Related concepts

• Kafka Connectpart of · Kafka Connect is a framework for reliably transferring data between Apache Kafka and external systems through configurable source and sink connectors.
• distributed modeused by · Distributed mode is a Kafka Connect operating mode in which multiple workers coordinate across a cluster to run connector tasks and store internal data.
• taskhas part · A Kafka Streams task is an independently executable processing unit that processes assigned input partitions through a topology and its associated state stores.
• heartbeatuses · A heartbeat is a periodic signal or message indicating that a component or system remains active and reachable.
• group coordinatorused by · A group coordinator is the Kafka broker responsible for managing a consumer group’s membership, liveness, partition assignments, and committed offsets.
• workerkind of · A worker is a person or computational execution unit assigned to perform a task as part of a larger system or activity.

This explainer was generated by Manning's knowledge graph from the text of Kafka Streams in Action, Second Edition. The section links point to the parts of the book it draws on.