Kafka Connect
Kafka Connect is a framework for reliably transferring data between Apache Kafka and external systems through configurable source and sink connectors.
Kafka for Architects
Kafka Connect is a framework for moving data between Kafka and external systems through configurable connectors rather than custom application code.
- How it is used
- Moves data between Kafka and external systems through configured pipelines instead of custom application producers and consumers.
- How deep it goes
- Explains plugin setup, connector deployment, client roles, internal topics, and operational pitfalls without walking through plugin implementation.
- Examples
- Uses address replication, an incrementing-ID transaction table, and a multi-source pipeline with database and cloud-storage sinks.
- What you can do after
- Operate/configure it by installing a suitable plugin, mapping external data to topics, and deploying a connector configuration through the REST API.
Apache Kafka in Action
Kafka Connect is a framework for reliably moving data between Kafka and external systems through connectors.
- How it is used
- Moves data between Kafka and existing databases, files, messaging systems, or other external systems without writing a separate producer or consumer for each.
- How deep it goes
- Explains source and sink roles, progress tracking, worker scaling, REST management, JDBC modes, and error handling through configuration and tests.
- Examples
- Tests an append-only file source with a console consumer and contrasts new-row capture with missed updates in a SQLite JDBC source.
- What you can do after
- Operate/configure it to move records through connectors, inspect status, scale tasks across workers, and choose error-handling behavior.
Kafka Streams in Action, Second Edition
Kafka Connect is a framework for importing data into Apache Kafka and exporting data from it through connectors and related processing components.
- How it is used
- Integrates existing external systems with Kafka by importing records through source connectors and exporting them through sink connectors.
- How deep it goes
- Explains connector roles, configuration, SMTs, deployment, and sink failure settings, while deferring custom connector implementation details.
- Examples
- Uses retail sales masking and dashboard export, plus PostgreSQL orientation registrations sent to Kafka and Elasticsearch.
- What you can do after
- Apply it via existing connectors by configuring source or sink data flows and lightweight record transformations.
Read how Kafka Streams in Action, Second Edition explains Kafka Connect →
Architecting an Apache Iceberg Lakehouse
Kafka Connect is a framework for continuously transferring data between Apache Kafka and external systems through connectors.
- How it is used
- Provides modular streaming ingestion from Kafka topics and external sources into Iceberg tables.
- How deep it goes
- Defines its ingestion role and architectural tradeoffs but does not detail connector configuration or implementation internals.
- Examples
- Uses market-tick data, order events, operational databases, APIs, and SaaS platforms as ingestion scenarios.
- What you can do after
- Apply connector-based streaming ingestion while separating append workloads from batch reconciliation and correction workflows.
Read how Architecting an Apache Iceberg Lakehouse explains Kafka Connect →
Related concepts
- connectoruses · A connector is a software element or mechanism that links components, applications, or resources to enable or represent their interaction, communication, or data exchange.
- Apache Kafkauses · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.
- sink connectoruses · A sink connector is a Kafka Connect connector that reads records from Kafka and exports them to an external destination system.
- distributed modeused by · Distributed mode is a Kafka Connect operating mode in which multiple workers coordinate across a cluster to run connector tasks and store internal data.
- topicuses · A topic is a named messaging destination or stream to which producers publish messages or records and from which consumers receive or read them.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.