Source connector

As explained in Apache Kafka in Action

A source connector is a Kafka Connect connector that reads data from an external source system and publishes it to Kafka.

A source connector reads records from an external system and publishes them to Kafka. It is the ingestion side of Kafka Connect, contrasting with a sink connector, which exports Kafka records to another system. Typical sources include databases, filesystems, and messaging systems. section 11.1

Figure 11.1 Kafka Connect integrates third-party systems with Kafka.

Use the integration diagram to trace data from a third-party system through a source connector into Kafka. section 11.1

A connector class selects the implementation, such as FileStreamSourceConnector, JdbcSourceConnector, or Debezium’s PostgresConnector. Configuration can split work across tasks with tasks.max, although the connector or source structure may limit the effective parallelism. Source data is modeled as partitioned streams so Kafka can scale processing while preserving each stream’s required ordering and tracking progress. section 11.6.1 section 11.1

Database ingestion

A JDBC source connector can read from a database and publish records to Kafka. Database tables can be treated as independent streams, with sequential identifier columns helping track which rows Kafka Connect has already seen. For database changes, a connector such as Debezium can implement change data capture. section 11.1 section 11.10

How Apache Kafka in Action covers it

How deep it goes

Defines the data flow and connector configuration, then addresses partitioning and task limits without detailing connector implementation internals.

Examples

Uses FileStreamSourceConnector, JDBC source connectors, database tables, sequential identifiers, and Debezium’s PostgresConnector.

What it assumes

Assumes Kafka concepts such as topics and partitions while distinguishing source connectors from sink connectors.

What you can do after

Operate/configure ingestion through connector classes, task limits, and Kafka Connect’s management interfaces.

How it connects

Connects source connectors with Kafka Connect, Kafka topics, partitions, offsets, sink connectors, and change data capture.

liveBookmanning.com

Read Apache Kafka in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Kafka Streams in Action, Second EditionSourceConnector is the Kafka Connect connector type used to configure tasks that import records from an external system into Kafka.
• 

  liveBookmanning.com
  Kafka for ArchitectsA source connector is a Kafka Connect connector that reads data from an external source system and writes it to Kafka.

Compare how 3 books explain source connector →

Related concepts

• connectorkind of · A connector is a software element or mechanism that links components, applications, or resources to enable or represent their interaction, communication, or data exchange.
• topicproduces · A topic is a named messaging destination or stream to which producers publish messages or records and from which consumers receive or read them.
• sink connectorcontrasts with · A sink connector is a Kafka Connect connector that reads records from Kafka and exports them to an external destination system.
• single message transformationuses · A single message transformation is a stateless Kafka Connect operation that independently modifies or filters each record passing through a connector.
• pollinguses · Polling is the repeated checking or querying of a resource, system, or condition, often at intervals, to detect data, events, or state changes.
• database tableuses · A database table is a relational database structure that organizes records into rows and defined columns.

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.