A new business requirement may add a field to an event that several services consume. Schema evolution is the controlled change of that event’s data contract: the producer must be able to publish the new structure without unexpectedly breaking consumers that still expect the old one. section 6.2.1
With Schema Registry, a changed schema is registered as a new, immutable version rather than altering an existing one. A producer puts the schema ID in each serialized message; a consumer uses that ID to retrieve the schema needed to deserialize it. Compatibility checks help govern which versions can coexist, but a schema ID does not replace communication between teams. section 1.5.1 section 6.3.4
Figure 6.12 Interactions with the Schema Registry. The schema is registered with the Schema Registry server, which stores it in Kafka. Producers use this schema to serialize messages, and consumers retrieve the schema from the Schema Registry to deserialize messages.
Trace how registration, producer serialization, and consumer schema retrieval work together when messages carry schema IDs rather than complete schemas. section 6.4.1
Treat a proposed change as both a contract decision and an operational change:
- Define the compatibility strategy for the event schema—backward, forward, or full—and record it with the topic’s other requirements. section 10.1.3
- Check compatibility, document the change, and notify dependent teams; consumers can migrate their code or, where appropriate, rely on backward compatibility. section 6.2.6 section 6.3.4
- Plan a migration when a change, such as renaming a field, changing its type, or restructuring an entity, does not fit the chosen compatibility constraints. section 6.4.6
When a database column changes
A source-system table gains a column, but a Kafka Connect pipeline still expects the earlier Avro schema. The source connector may fail to write the incompatible message and stop; source connectors lack dead-letter topic support. At the sink, an incompatible message can instead be redirected to a dead-letter topic, which still requires monitoring and alerts. Field-by-field replication makes coordinated changes across systems particularly important. section 7.3.10
Central registration does not validate every message at the Kafka broker: clients remain responsible for using schemas correctly. A producer-owned, accessible contract repository and a way for consumers to propose changes help keep the technical schema aligned with the people and services that depend on it. section 6.4.6 section 6.3.4
