A Kafka serializer converts an application object into the byte array Kafka stores in a topic and transports over the network. A producer calls serialize on a Serializer implementation before sending a record; a consumer uses the matching deserializer to reconstruct the object. Producers and consumers are independent of the implementation, so the serialized object type becomes part of their implicit event contract. section 3.1
Figure 3.1 The conversion of objects to bytes and bytes to objects happens at the client level.
Follow the boundary: application objects become bytes inside the producer client, cross Kafka as bytes, and become objects inside the consumer client. section 3.1
Configure serializer classes on the Kafka producer rather than calling serialization code from application logic. The producer only requires an implementation of the Serializer interface and receives bytes from its serialize method. A matching consumer configuration supplies a Deserializer; both sides must agree on the event structure for the topic. section 3.2.9 section 3.7
Choose a serialization approach based on how event contracts are managed:
- Use Schema Registry-aware serializers for Avro, Protobuf, or JSON Schema when schemas should be centrally stored, validated, versioned, and discovered by clients. The serializer retrieves and locally caches schemas while serializing records. section 3.2.1 section 3.7
- Implement org.apache.kafka.common.serialization.Serializer when Schema Registry is unavailable or unwanted. Implement serialize; optionally override configure and close, then provide the class name in producer configuration. section 3.8
- Use a Serde in Kafka Streams, where a paired serializer and deserializer is needed because the embedded producer and consumer are not directly accessible. section 3.7
Custom JSON serializer
A custom serializer can return null for null data, then delegate conversion to Jackson's objectMapper.writeValueAsBytes(data). Convert JsonProcessingException into SerializationException so serialization failures are reported through Kafka's expected error type. Configure this implementation on the producer as the value serializer. section 3.8 section 3.2.9
Schema Registry removes the need to write common serialization code and standardizes formats across teams. Its serializers work with schemas as the source of truth, adding validation and schema-evolution support. Without it, custom serializers still work because Kafka clients are decoupled from serializer implementations, but schema validation checks are lost. section 3.2.1 section 3.8
For an Avro event, generate Java classes from a registered schema, build a typed domain object, configure KafkaAvroSerializer and the Schema Registry URL, then produce the object. A consumer configured with KafkaAvroDeserializer can receive a specific generated record or a GenericRecord depending on its reader setting. section 3.2.9
Serializer configuration matters when one topic carries multiple event types. With an Avro union schema referenced from individual event schemas, disable automatic schema registration and enable use of the latest schema version; otherwise an individual event schema can overwrite the union or schema lookup can fail. section 3.6
