liveBook · Manning

Incrementing mode

As explained in Apache Kafka in Action

Incrementing mode is a JDBC source connector polling mode that detects new rows using a strictly increasing column value.

Incrementing mode is a JDBC source connector polling mode for tables with a strictly increasing column, usually an ID. The connector stores its latest value and polls for rows above that value, allowing newly inserted records to reach Kafka. section 11.8.3

Detecting a new customer

With mode=incrementing and incrementing.column.name=ID, a connector can process IDs 1 and 2, then publish a row inserted with ID 3. If the last processed ID was 3, its next query is equivalent to SELECT * FROM customers WHERE id > 3. Kafka Connect converts the resulting database row into a JSON message. section 11.8.3

The mode tracks new IDs rather than changes to existing rows. Updating the row with ID 3 therefore produces no message. Deletions are not detected either, because they create no new incrementing value. section 11.8.3

Use incrementing mode when:

  • The selected column increases monotonically and strictly; reused or smaller IDs can be skipped. section 11.8.3
  • New-row capture is sufficient and updates or deletes do not need to be propagated. section 11.8.3
  • Changes to existing rows must be captured through a timestamp column; use timestamp mode or timestamp+incrementing instead. section 11.8.3 section 11.10

How Apache Kafka in Action covers it

How deep it goes
Explains stored offsets, generated queries, insert behavior, and monotonicity failures, but stops short of connector deployment details.
Examples
Demonstrates customer inserts, an existing-row update, and an ID-filtered SQL query.
What it assumes
Assumes a database table, Kafka Connect JDBC source configuration, and a column whose values increase strictly.
What you can do after
Apply it via a Kafka Connect JDBC source connector to publish newly inserted rows while recognizing missed updates and deletions.
How it connects
Contrasts incrementing mode with timestamp mode, timestamp+incrementing mode, and change data capture.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.