Distributed mode

As explained in Kafka Streams in Action, Second Edition

Distributed mode is a Kafka Connect operating mode in which multiple workers coordinate to run connector tasks across a cluster.

Distributed mode runs Kafka Connect with multiple workers coordinating connector tasks across a cluster. It suits production environments because work can run in parallel and recover from worker failures. section 5.3

A distributed Kafka Connect deployment separates these responsibilities:

• Workers provide the Connect runtime and receive server-level configuration such as converters, plugin paths, and cluster identifiers. section 5.3
• Connector configurations define individual integrations and can override worker defaults, including key and value converters. section 5.3
• Tasks are spread across connector instances instead of remaining on one worker, enabling cluster-level parallelism and fault tolerance. section 5.3

Starting a source connector

A JDBC source connector can be started through the Connect REST API with a PostgreSQL connection, timestamp-based polling, JSON value conversion, and a maximum task count. section 5.3

Figure 5.4 Connect in distributed mode; tasks get spread around to other connector instances.

Shows how distributed mode spreads connector tasks across Connect instances. section 5.3

Figure 5.5 Connect in distributed mode provides fault tolerance.

Shows the fault-tolerance benefit of distributed task execution. section 5.3

How Kafka Streams in Action, Second Edition covers it

How deep it goes

Explains worker and connector configuration plus task distribution, but does not detail coordination internals.

Examples

Uses Docker Compose setup, a JDBC source connector, PostgreSQL, the Connect REST API, and converter configuration.

What you can do after

Operate Kafka Connect in a clustered production deployment and configure integrations through its REST API.

How it connects

Connects Kafka Connect with standalone mode, connect workers, connector tasks, converters, and Kafka brokers.

liveBookmanning.com

Read Kafka Streams in Action, Second Edition on liveBook Get the book at manning.com

Related concepts

• standalone modecontrasts with · Standalone mode is an operating configuration in which a single worker or component runs independently rather than as part of a coordinated cluster.
• Kafka Connectuses · Kafka Connect is a framework for reliably transferring data between Apache Kafka and external systems through configurable source and sink connectors.
• Kafka Connect workeruses · A Kafka Connect worker is a process that manages connectors and runs their tasks within a Kafka Connect cluster.
• parallelismuses · Parallelism is the concurrent execution of multiple computational operations, often across multiple processing resources.

This explainer was generated by Manning's knowledge graph from the text of Kafka Streams in Action, Second Edition. The section links point to the parts of the book it draws on.