Schema evolution
Schema evolution is the controlled modification of a data schema over time while managing compatibility with existing data, producers, consumers, or stored records.
Kafka for Architects
A schema change is a modification to the structure, fields, types, or format of data exchanged or stored by a system.
- How it is used
- Manages changes to Kafka event contracts without unexpectedly disrupting producers, consumers, or data pipelines.
- How deep it goes
- Explains versioning, schema-ID exchange, compatibility planning, team coordination, and connector failure modes without a low-level implementation walkthrough.
- Examples
- Uses an added event attribute, a changed database column, and field renaming or type changes to expose migration risks.
- What you can do after
- Operate/configure it by recording a compatibility strategy, checking proposed versions, notifying consumers, and monitoring connector failures.
Architecting an Apache Iceberg Lakehouse
Schema evolution is the controlled modification of a dataset schema over time while preserving compatibility with existing data and readers.
- How it is used
- Guides lakehouse ingestion, governance, and table design as source systems and business requirements change.
- How deep it goes
- Explains field IDs, validation, mappings, registries, and operational safeguards, but does not provide executable implementation commands.
- Examples
- Uses evolving mobile events, high-throughput streams, batch pipelines, variant columns, and an audit of manual schema coordination.
- What you can do after
- Operate/configure ingestion pipelines that validate, map, register, and monitor schema changes before they disrupt downstream consumers.
Read how Architecting an Apache Iceberg Lakehouse explains schema evolution →
Kafka Streams in Action, Second Edition
Schema evolution is the controlled modification of a data schema while preserving compatibility for existing producers, consumers, or stored records.
- How it is used
- Governs message-structure changes between Kafka producers and consumers through centralized schemas and compatibility checks.
- How deep it goes
- Explains subjects, naming strategies, compatibility modes, transitive checks, caching, and serializer integration.
- Examples
- Uses an evolving AvroAvenger record, generated Java classes, schema registration commands, and Avro producer-consumer configuration.
- What you can do after
- Apply it via Schema Registry to validate, register, serialize, deserialize, and test compatible Kafka schema changes.
Read how Kafka Streams in Action, Second Edition explains schema evolution →
Apache Kafka in Action
Schema versioning is the management of successive schema definitions as data structures evolve over time.
- How it is used
- Keeps Kafka producers and consumers interoperable as data structures change over time.
- How deep it goes
- Explains version coexistence, registry identifiers, and compatibility checks, but does not detail migration procedures.
- Examples
- Uses a supermarket Product record with an optional field, binary payload sizing, and schema IDs in Kafka messages.
- What you can do after
- Apply it via a schema registry to track message schemas and let producers and consumers evolve independently.
Related concepts
- schema registryused by · A schema registry is a service that stores, versions, and provides schemas for structured data, often validating compatibility between schema versions.
- Apache Icebergused by · Apache Iceberg is an open table format for managing large analytical datasets with metadata, schema evolution, snapshots, and transactional operations across distributed or object storage.
- environment variableconfigured by · An environment variable is a named value supplied through an operating-system or process environment and made available to programs for runtime configuration or information.
- source connectorused by · A source connector is a Kafka Connect connector that reads data from an external source system and publishes records to Kafka.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.