liveBook · Manning

Source partition

As explained in Kafka Streams in Action, Second Edition

A source partition is a logical description of the portion or location of an external system from which a connector obtains data.

A source partition describes the general location from which a connector obtains data. In Kafka Connect, it is represented as a Map and supplied to the SourceRecord constructor alongside a source offset. section 5.6.1

Use the source partition to identify the source location, and the source offset to identify progress within that location:

  • Identify a database table, file, or API URL as the source partition. section 5.6.1
  • Record the individual record’s position in the retrieved result as the source offset. section 5.6.1
  • Pass both Maps when constructing each SourceRecord; the record may carry the generated schema into Kafka. section 5.6.1

Ticker API connector

A source task polls a ticker API, flattens its JSON results, and creates one SourceRecord per entry. It uses the API URL as the source partition and the entry’s position in the result as the source offset. section 5.6.1

How Kafka Streams in Action, Second Edition covers it

How deep it goes
Defines the partition as a location descriptor, explains its Map representation, and distinguishes it from the record-level source offset.
Examples
Uses a database table, file name, and ticker API URL as source partitions in connector scenarios.
What you can do after
Build Kafka Connect SourceRecord metadata by assigning an external source location and per-record position.
How it connects
Connects source partitions with SourceRecord, source offsets, source connectors, schemas, and Kafka topics.
Cover of Kafka Streams in Action, Second Edition
liveBookmanning.com
Read Kafka Streams in Action, Second Edition on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Kafka Streams in Action, Second Edition. The section links point to the parts of the book it draws on.