liveBook · Manning

Worker configuration

As explained in Apache Kafka in Action

A worker configuration is a set of settings that controls Kafka Connect worker resources, task distribution, and integration behavior.

Worker configuration defines how Kafka Connect workers operate across a cluster. It applies to every connector running on those workers, while connector configuration is supplied separately through the REST API. section 11.4

A worker configuration coordinates the cluster through several settings:

  • bootstrap.servers identifies the Kafka cluster used by the workers. section 11.4
  • group.id gives all workers in the same Kafka Connect cluster a shared identity; it is not a traditional consumer-group identifier. section 11.4
  • config.storage.topic stores connector configurations, status.storage.topic stores connector and task status, and offset.storage.topic stores source-connector offsets. section 11.4
  • key.converter and value.converter define message encoding and decoding. Supported converter families include String, Avro, Protobuf, JSON Schema, simple JSON, and byte arrays; schema-based converters require Schema Registry. section 11.4

Minimal worker.properties

bootstrap.servers=localhost:9092 group.id=connect config.storage.topic=connect-config offset.storage.topic=connect-offset status.storage.topic=connect-status key.converter=org.apache.kafka.connect.storage.StringConverter value.converter=org.apache.kafka.connect.storage.StringConverter plugin.path=<PATH-TO-YOUR-USER-DIRECTORY>/kafka/libs/ This file supplies shared worker settings at startup. Connector-specific settings can then be managed through the REST API. section 11.4

Worker and connector settings support operational behavior such as resource allocation, task distribution, resilience, error tolerance, logging, retries, and dead-letter handling. Single Message Transformations can rename, mask, or reposition values, but they are intended for simple changes rather than complex ETL. section 11.10 section 11.10

bootstrap.servers=localhost:9092
group.id=connect
config.storage.topic=connect-config
offset.storage.topic=connect-offset
status.storage.topic=connect-status
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
plugin.path=<PATH-TO-YOUR-USER-DIRECTORY>/kafka/libs/

Browse the compact worker.properties listing to compare the cluster identity, internal topics, converters, and plugin path in one configuration. section 11.4

How Apache Kafka in Action covers it

How deep it goes
Explains worker-versus-connector scope and key properties with a runnable listing, while deferring connector-specific configuration details.
Examples
Uses a worker.properties listing with bootstrap servers, cluster identity, internal topics, StringConverter settings, and a plugin path.
What you can do after
Operate/configure Kafka Connect workers using a properties file and manage connector settings through the REST API.
How it connects
Connects Kafka Connect workers with connectors, REST API, internal Kafka topics, converters, and Schema Registry.
Cover of Apache Kafka in Action
liveBookmanning.com
Read Apache Kafka in Action on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.