liveBook · Manning

Kafka Connect converter

As explained in Apache Kafka in Action

JsonConverter is a Kafka Connect converter that serializes and deserializes record data as JSON, optionally including schemas.

A Kafka Connect converter serializes and deserializes record keys or values. JsonConverter writes JSON and can include a schema with each message; disabling schemas produces simpler JSON. section 11.4 section 11.8.2

Configure converters deliberately because the choice affects message format and operational behavior:

  • Set key.converter and value.converter to select separate key and value encoders. section 11.4
  • Use value.converter.schemas.enable to include or omit schemas in JsonConverter output. section 11.4 section 11.8.2
  • Prefer a schema-registry converter such as AvroConverter when per-message JSON schemas become excessively verbose. section 11.4
  • Keep converter settings in worker configuration even when a connector overrides them. section 11.4

JDBC source with simple JSON

A JDBC source connector can override the worker's value converter with org.apache.kafka.connect.json.JsonConverter and set value.converter.schemas.enable to false. With table.whitelist set to customers and topic.prefix set to sqlite_, records go to sqlite_customers as simple JSON. section 11.8.2

{
  "name": "customers-jdbc-source-connector",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.url": "jdbc:sqlite:/path/to/customers.db",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",   
    "value.converter.schemas.enable": false,
    "table.types": "TABLE",
    "table.whitelist": "customers",
    "topic.prefix": "sqlite_",
    "mode": "incrementing",
    "incrementing.column.name": "id",
    "batch.size": 1000
  }
}

Inspect a complete connector configuration showing JsonConverter, disabled schemas, table selection, topic naming, and incrementing JDBC capture. section 11.8.2

How Apache Kafka in Action covers it

How deep it goes
Explains converter selection and JSON schema toggling, then contrasts verbose embedded schemas with schema-registry alternatives.
Examples
Uses a customers JDBC source, simple JSON without schemas, and the sqlite_customers topic.
What it assumes
Assumes Kafka Connect worker and connector configuration, while demonstrating the converter settings needed for a JDBC source.
What you can do after
Operate/configure Kafka Connect with JSON records and choose whether each message carries an embedded schema.
How it connects
Ties JsonConverter to Kafka Connect workers, JDBC Source Connector, StringConverter, AvroConverter, and Schema Registry.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.