liveBook · Manning

Connector configuration

As explained in Apache Kafka in Action

Connector configuration is the set of named parameters that defines how a Kafka Connect connector connects to and processes an external system.

Connector configuration is the named set of parameters that defines how a Kafka Connect connector connects to and processes an external system. Unlike worker configuration, which applies to every connector on a worker, connector configuration targets one connector and is submitted through the REST API. Kafka Connect distributes these configurations through its internal configuration topic. section 11.4

Create and change a file source connector

A connector definition names the connector and supplies connector-specific settings such as connector.class, tasks.max, file, and topic. POSTing this JSON to /connectors creates customers-source. If the source file changes, PUT the revised configuration to /connectors/customers-source/config, replacing /tmp/customers.txt with /tmp/new_customers.txt. section 11.5.2

Manage connector configuration and runtime behavior through the REST API:

  • Create a connector with POST /connectors and a JSON object containing its name and connector-specific configuration. section 11.5.2
  • Modify settings with PUT /connectors/<connector>/config, such as changing the source file. section 11.5.2
  • Pause connectors during third-party maintenance, resume them afterward, and restart failed connectors or tasks after resolving persistent problems. section 11.5.2
bootstrap.servers=localhost:9092
group.id=connect
config.storage.topic=connect-config
offset.storage.topic=connect-offset
status.storage.topic=connect-status
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
plugin.path=<PATH-TO-YOUR-USER-DIRECTORY>/kafka/libs/

Use the worker configuration listing to see the surrounding Kafka Connect cluster settings, including bootstrap servers, storage topics, converters, and plugin path. section 11.4

How Apache Kafka in Action covers it

How deep it goes
Explains scope, storage, creation, modification, pausing, resuming, and restarting, but does not detail every connector-specific parameter.
Examples
Uses a customers-source FileStreamSource reading customer files into the customers topic, then changes its file parameter.
What it assumes
Assumes familiarity with Kafka Connect workers and introduces connector-specific configuration through concrete REST API operations.
What you can do after
Operate connectors by creating, updating, pausing, resuming, deleting, and restarting them through documented REST endpoints.
How it connects
Connects worker configuration with REST API, internal configuration topics, source connectors, tasks, converters, and Kafka Connect clusters.
Cover of Apache Kafka in Action
liveBookmanning.com
Read Apache Kafka in Action on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.