A new Kafka consumer may need to read product records written months ago. Backward compatibility means a consumer using the current schema can read those older records. This matters because records can remain in a topic while producers and consumers use different schema versions. section 13.1.2 section 13.1.1
Removing a product field
Suppose product records contain a stock field that is removed from the new schema. Update the consumer to ignore stock first; it can then read older records that still contain the field. Only afterward should the producer stop writing stock. An older consumer that still expects stock may fail on new records. section 13.1.2
The direction is important: backward compatibility protects current consumers reading old data, not old consumers reading new data. Renaming fields can break both directions. If a necessary change cannot preserve compatibility, it may require a new topic, migration of old data, and migration of producers and consumers. section 13.1.2 section 13.1.1
Figure 13.2 We consider schema compatibility from the consumer’s perspective and which schema versions they can understand. If we can’t or don’t want to guarantee any compatibility, a consumer can only understand their own schema version. A backward-compatible consumer can read historical data but can’t handle newer data. A forward-compatible consumer can read new data, but not historical data. The optimum is a consumer that can read both historical and new data, which is fully compatible in both directions.
The diagram contrasts which schema versions a consumer can read under backward, forward, full, or no compatibility. section 13.1.2
A schema registry can version schemas and check proposed changes against older versions. Plan its availability and configuration carefully: it adds a system component, while Kafka itself accepts byte arrays and schema checks occur on the client side. section 13.1.3
