A serializer converts structured producer data into bytes for Kafka. Brokers transfer binary data without interpreting message content, so producers and consumers must agree on the corresponding serialization and deserialization formats. section 3.1.4
Kafka producer serialization has several architectural implications:
- Keys and values are serialized independently and may use different serializer implementations. section 3.1.4 section 3.2
- The consumer must configure matching deserializers; Kafka does not provide a registry that verifies client agreement. section 3.1.4
- Primitive serializers support types such as strings, integers, longs, and byte arrays; custom implementations are also possible. section 3.1.4
- A format mismatch produces an error at runtime, making serializer configuration an operational reliability concern. section 3.1.4
Separate key and value serializers
A producer can encode message keys as integers and values as strings: key.serializer=org.apache.kafka.common.serialization.IntegerSerializer and value.serializer=org.apache.kafka.common.serialization.StringSerializer. The consumer then selects the corresponding integer and string deserializers. section 3.1.4
Serialization occurs after a producer message is created and before it enters a memory batch. Kafka’s producer library then handles batching, dispatch, replication, acknowledgments, and retry behavior. section 3.2
In Kafka Connect, converters perform the serializer/deserializer role between ConnectRecord objects and Kafka bytes. Source connectors serialize records into formats such as Avro, JSON, or Protobuf; sink connectors convert Kafka data back into ConnectRecords. section 7.3.3
key.serializer=org.apache.kafka.common.serialization.IntegerSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer
Inspect the compact producer configuration showing independent key and value serializer selection. section 3.1.4
