Schema evolution manages successive data-structure definitions so Kafka producers and consumers continue communicating as shared data changes. Without governance, inconsistent product data can cause inventory errors or incorrect pricing. section 13.0 section 13.4
A schema registry centralizes schemas and supports compatibility checks before updates. Producers and consumers can then evolve independently while incompatible changes are prevented from disrupting data processing. section 13.4
Avro provides a practical pattern for traceable schema evolution:
- Each Avro message combines a schema with its payload, making the data self-describing. section 13.1.4
- Because Kafka topics can retain data for a long time, multiple schema versions may coexist. section 13.1.4
- Instead of repeating a large JSON schema in every message, store it in the registry and send only its schema ID. section 13.1.4
Evolving a product record
A supermarket Product schema can define id, name, price, and stock, then add an optional origin field with a null default. A later message may also contain a category field; registry tracking associates each message with the schema needed to interpret it. section 13.1.4
{
"type": "record",
"name": "Product",
"namespace": "com.example.supermarket",
"fields": [
{"name": "id", "type": "string"},
{"name": "name", "type": "string"},
{"name": "price", "type": "float"},
{"name": "stock", "type": "int"},
{"name": "origin", "type": ["null", "string"], "default": null}
]
}
Inspect the Avro Product record to see how fields and an optional value are declared in JSON. section 13.1.4
