Kafka Connect
Kafka Connect is a framework for reliably transferring data between Apache Kafka and external systems through configurable source and sink connectors.
Kafka for Architects
Kafka Connect is a framework for moving data between Kafka and external systems through configurable connectors rather than custom application code.
- How it is used
- Moves data between Kafka and external systems through configured pipelines instead of custom application producers and consumers.
- How deep it goes
- Explains plugin setup, connector deployment, client roles, internal topics, and operational pitfalls without walking through plugin implementation.
- Examples
- Uses address replication, an incrementing-ID transaction table, and a multi-source pipeline with database and cloud-storage sinks.
- What you can do after
- Operate/configure it by installing a suitable plugin, mapping external data to topics, and deploying a connector configuration through the REST API.
Apache Kafka in Action
Kafka Connect is a framework for reliably moving data between Kafka and external systems through connectors.
- How it is used
- Moves data between Kafka and existing databases, files, messaging systems, or other external systems without writing a separate producer or consumer for each.
- How deep it goes
- Explains source and sink roles, progress tracking, worker scaling, REST management, JDBC modes, and error handling through configuration and tests.
- Examples
- Tests an append-only file source with a console consumer and contrasts new-row capture with missed updates in a SQLite JDBC source.
- What you can do after
- Operate/configure it to move records through connectors, inspect status, scale tasks across workers, and choose error-handling behavior.
Kafka Streams in Action, Second Edition
Kafka Connect is a framework for importing data into Apache Kafka and exporting data from it through connectors and related processing components.
- How it is used
- Integrates existing external systems with Kafka by importing records through source connectors and exporting them through sink connectors.
- How deep it goes
- Explains connector roles, configuration, SMTs, deployment, and sink failure settings, while deferring custom connector implementation details.
- Examples
- Uses retail sales masking and dashboard export, plus PostgreSQL orientation registrations sent to Kafka and Elasticsearch.
- What you can do after
- Apply it via existing connectors by configuring source or sink data flows and lightweight record transformations.
Read how Kafka Streams in Action, Second Edition explains Kafka Connect →
Architecting an Apache Iceberg Lakehouse
Kafka Connect is a framework for continuously transferring data between Apache Kafka and external systems through connectors.
- How it is used
- Provides modular streaming ingestion from Kafka topics and external sources into Iceberg tables.
- How deep it goes
- Defines its ingestion role and architectural tradeoffs but does not detail connector configuration or implementation internals.
- Examples
- Uses market-tick data, order events, operational databases, APIs, and SaaS platforms as ingestion scenarios.
- What you can do after
- Apply connector-based streaming ingestion while separating append workloads from batch reconciliation and correction workflows.
Read how Architecting an Apache Iceberg Lakehouse explains Kafka Connect →
What to know first
- topicA topic is a named channel or category that groups messages or records for publication and delivery to interested consumers.
- clientA client is a person, organization, device, or software component that receives or uses a product, service, resource, or functionality provided by another party or system.
- stream processingStream processing is the incremental processing of data items as they become available, without requiring the complete dataset to be present first.
- eventA software event is an occurrence, state change, or notification that a system can detect, record, communicate, or handle.
Related concepts
- Apache Kafkauses · Apache Kafka is a distributed event-streaming platform for publishing, storing, and processing streams of records.
- connectoruses · A connector is a software element or mechanism that links components, applications, or resources to enable or represent their interaction, communication, or data exchange.
- Apache MirrorMakerused by · Apache MirrorMaker is an Apache Kafka tool that replicates topics, consumer offsets, and related metadata between Kafka clusters.
- distributed modeused by · Distributed mode is a Kafka Connect operating mode in which multiple workers coordinate across a cluster to run connector tasks and store internal data.
- Elasticsearchuses · Elasticsearch is a distributed search and analytics engine for indexing, querying, and analyzing structured and unstructured data.
- errors.toleranceconfigures · errors.tolerance is a Kafka Connect setting that determines whether connector processing continues after record-processing errors.
Questions about Kafka Connect
What is Kafka Connect?
Kafka Connect is a framework for reliably transferring data between Apache Kafka and external systems through configurable source and sink connectors.
Which Manning books explain Kafka Connect?
Kafka for Architects, Apache Kafka in Action, Kafka Streams in Action, Second Edition and Architecting an Apache Iceberg Lakehouse.
What should I know before learning Kafka Connect?
Manning books that teach Kafka Connect usually introduce topic, client, stream processing and event first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



