Source connector
A source connector is a Kafka Connect connector that reads data from an external source system and publishes records to Kafka.
Kafka Streams in Action, Second Edition
SourceConnector is the Kafka Connect connector type used to configure tasks that import records from an external system into Kafka.
- How it is used
- Imports external records into Kafka topics through configurable Kafka Connect plugins.
- How deep it goes
- Explains connector configuration, task generation, task partitioning, and polling, while deferring parts of the complete implementation.
- Examples
- Uses stock ticker API ingestion, JDBC PostgreSQL configuration, and masked retail sales transactions.
- What you can do after
- Build it from scratch by extending SourceConnector, defining configuration, and generating task configurations for external-source ingestion.
Read how Kafka Streams in Action, Second Edition explains source connector →
Kafka for Architects
A source connector is a Kafka Connect connector that reads data from an external source system and writes it to Kafka.
- How it is used
- Moves external-system data into Kafka topics through configured integration rather than a custom producer.
- How deep it goes
- Explains polling, task parallelism, source offsets, converters, and deployment through a worked JDBC configuration.
- Examples
- Uses customer address replication, reference tables, and a JDBC transaction-table configuration with an incrementing ID.
- What you can do after
- Operate/configure it by installing a plugin, setting source and progress parameters, and deploying a connector through the REST API.
Apache Kafka in Action
A source connector is a Kafka Connect connector that reads data from an external source system and publishes it to Kafka.
- How it is used
- Moves records from databases, files, or other external systems into Kafka through Kafka Connect.
- How deep it goes
- Defines the data flow and connector configuration, then addresses partitioning and task limits without detailing connector implementation internals.
- Examples
- Uses FileStreamSourceConnector, JDBC source connectors, database tables, sequential identifiers, and Debezium’s PostgresConnector.
- What you can do after
- Operate/configure ingestion through connector classes, task limits, and Kafka Connect’s management interfaces.
Related concepts
- connectorkind of · A connector is a software element or mechanism that links components, applications, or resources to enable or represent their interaction, communication, or data exchange.
- topicproduces · A topic is a named messaging destination or stream to which producers publish messages or records and from which consumers receive or read them.
- sink connectorcontrasts with · A sink connector is a Kafka Connect connector that reads records from Kafka and exports them to an external destination system.
- single message transformationuses · A single message transformation is a stateless Kafka Connect operation that independently modifies or filters each record passing through a connector.
- pollinguses · Polling is the repeated checking or querying of a resource, system, or condition, often at intervals, to detect data, events, or state changes.
- database tableuses · A database table is a relational database structure that organizes records into rows and defined columns.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.