Converter

As explained in Kafka Streams in Action, Second Edition

A converter is a Kafka Connect component that changes data between Connect's internal representation and a serialization format used by Kafka or an external system.

A converter changes data between Kafka Connect’s internal representation and the serialization format used by Kafka or an external system. In a source connector, it serializes records before they enter Kafka; in a sink connector, it deserializes records after they leave Kafka. section 5.3

Configure converters at the worker or connector level:

• Set key.converter to control serialization of record keys, such as org.apache.kafka.connect.storage.StringConverter. section 5.3
• Set value.converter to control serialization of record values, such as org.apache.kafka.connect.json.JsonConverter. section 5.3
• Treat worker settings as defaults: an individual connector can override them. section 5.3
• Configure related options such as value.converter.schemas.enable when the selected converter supports them. section 5.3

Postgres source connector

A JDBC source connector can use org.apache.kafka.connect.json.JsonConverter for values and set value.converter.schemas.enable to false. The connector then reads database records and serializes their values for Kafka. section 5.3

A converter is distinct from a Single Message Transform (SMT). An SMT performs a lightweight, one-record change between the connector and converter: before conversion for a source connector and after conversion for a sink connector. Use Kafka Streams or ksqlDB for joins, aggregations, or other complex operations. section 5.4

Figure 5.2 Connect converter changes the data format before the data gets into Kafka or after it leaves Kafka.

Follow the data path to see where conversion occurs before Kafka receives a record or after a record leaves Kafka. section 5.3

How Kafka Streams in Action, Second Edition covers it

How deep it goes

Defines conversion direction and key/value settings, but stops short of implementing a custom converter.

Examples

Uses StringConverter for keys, JsonConverter for values, and a JDBC source connector with schemas disabled.

What you can do after

Operate/configure Kafka Connect converters through worker defaults, connector overrides, and key/value serialization settings.

How it connects

Connects converters with source connectors, sink connectors, SMTs, Kafka Streams, and ksqlDB.

liveBookmanning.com

Read Kafka Streams in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Java Persistence with Spring Data and Hibernate epubA custom type converter is application-defined conversion logic that translates values between a domain type and a persistence representation.
• 

  liveBookmanning.com
  Spring in Action, Sixth EditionA Spring Converter is a class implementing the Converter interface to transform a value from one type into another during data binding.
• 

  liveBookmanning.com
  ASP.NET Core in Action, 3eA type converter is a .NET conversion component or convention that converts values between specified types, including converting textual input to another type.

Compare how 4 books explain converter →

Related concepts

• source connectorused by · A source connector is a Kafka Connect connector that reads data from an external source system and publishes records to Kafka.
• single message transformationused by · A single message transformation is a stateless Kafka Connect operation that independently modifies or filters each record passing through a connector.
• sink connectorused by · A sink connector is a Kafka Connect connector that reads records from Kafka and exports them to an external destination system.
• to_symincludes · to_sym is a Ruby conversion method that returns a symbol corresponding to a value's textual representation.
• to_sincludes · to_s is a Ruby conversion method that returns a string representation of its receiver.

This explainer was generated by Manning's knowledge graph from the text of Kafka Streams in Action, Second Edition. The section links point to the parts of the book it draws on.