A schema makes the data contract between Kafka producers and consumers explicit. It describes an event object independently of programming language: its name, fields, and each field’s type. Producers serialize objects to bytes, consumers deserialize bytes back to objects, and the schema defines the structure both sides expect. section 3.1 section 3.2
Kafka brokers store byte arrays without interpreting their data types. That decoupling lets independent applications exchange events, but it also makes an explicit schema valuable for ensuring that serialized records have the expected shape. section 3.2 section 3.0
An Avro event record
An Avro schema can define an Avenger record with string name and real_name fields plus a movies array whose default is an empty list. The schema file becomes the source for generated Java domain objects, reducing repeated object boilerplate and field-type mistakes. section 3.2.5 section 3.2.6
Schemas support an application workflow that keeps event structure deliberate:
- Register a schema under a subject through the Schema Registry REST API or a Gradle task. section 3.2.5 section 3.2.7
- Generate Java objects from Avro or Protobuf schemas, then create event instances that conform to their fields. section 3.2.6 section 3.2.9
- Configure KafkaAvroSerializer and KafkaAvroDeserializer with the Schema Registry URL to produce and consume those objects. section 3.2.9
- Evolve registered schemas with compatibility settings and versions; use transitive modes when compatibility must span all versions. section 3.9
Schema Registry centrally stores schemas, provides integrated serializers and deserializers, and supports Avro, Protocol Buffers, and JSON Schema. Its local serializer/deserializer caches avoid registry lookups when a schema is already available. Without Schema Registry, applications can instead supply custom Serializer and Deserializer implementations. section 3.8 section 3.9
A topic can carry related event types while remaining constrained. Schema references let a wrapper or union schema refer to allowed event schemas; with TopicNameStrategy, the topic is governed by one subject while permitting only those referenced types. section 3.6
Listing 3.2 Avro schema for Avengers
{"namespace": "bbejeck.chapter_3", #1
"type": "record", #2
"name": "Avenger", #3
"fields": [ #4
{"name": "name", "type": "string"},
{"name": "real_name", "type": "string"}, #5
{"name": "movies", "type":
{"type": "array", "items": "string"},
"default": [] #6
}
]
}
Inspect the Avro record structure directly: namespace, record name, fields, array type, and default value. section 3.2.5
