Schema evolution
Schema evolution is the controlled modification of a data schema over time while managing compatibility with existing data, producers, consumers, or stored records.
Kafka for Architects
A schema change is a modification to the structure, fields, types, or format of data exchanged or stored by a system.
- How it is used
- Manages changes to Kafka event contracts without unexpectedly disrupting producers, consumers, or data pipelines.
- How deep it goes
- Explains versioning, schema-ID exchange, compatibility planning, team coordination, and connector failure modes without a low-level implementation walkthrough.
- Examples
- Uses an added event attribute, a changed database column, and field renaming or type changes to expose migration risks.
- What you can do after
- Operate/configure it by recording a compatibility strategy, checking proposed versions, notifying consumers, and monitoring connector failures.
Architecting an Apache Iceberg Lakehouse
Schema evolution is the controlled modification of a dataset schema over time while preserving compatibility with existing data and readers.
- How it is used
- Guides lakehouse ingestion, governance, and table design as source systems and business requirements change.
- How deep it goes
- Explains field IDs, validation, mappings, registries, and operational safeguards, but does not provide executable implementation commands.
- Examples
- Uses evolving mobile events, high-throughput streams, batch pipelines, variant columns, and an audit of manual schema coordination.
- What you can do after
- Operate/configure ingestion pipelines that validate, map, register, and monitor schema changes before they disrupt downstream consumers.
Read how Architecting an Apache Iceberg Lakehouse explains schema evolution →
Kafka Streams in Action, Second Edition
Schema evolution is the controlled modification of a data schema while preserving compatibility for existing producers, consumers, or stored records.
- How it is used
- Governs message-structure changes between Kafka producers and consumers through centralized schemas and compatibility checks.
- How deep it goes
- Explains subjects, naming strategies, compatibility modes, transitive checks, caching, and serializer integration.
- Examples
- Uses an evolving AvroAvenger record, generated Java classes, schema registration commands, and Avro producer-consumer configuration.
- What you can do after
- Apply it via Schema Registry to validate, register, serialize, deserialize, and test compatible Kafka schema changes.
Read how Kafka Streams in Action, Second Edition explains schema evolution →
Apache Kafka in Action
Schema versioning is the management of successive schema definitions as data structures evolve over time.
- How it is used
- Keeps Kafka producers and consumers interoperable as data structures change over time.
- How deep it goes
- Explains version coexistence, registry identifiers, and compatibility checks, but does not detail migration procedures.
- Examples
- Uses a supermarket Product record with an optional field, binary payload sizing, and schema IDs in Kafka messages.
- What you can do after
- Apply it via a schema registry to track message schemas and let producers and consumers evolve independently.
Also covered in
- First-Party Data Activation
- Contract Testing in Action
- ScyllaDB in Action
- Building User-Friendly DSLs
- Akka in Action, Second Edition
- Java Persistence with Spring Data and Hibernate epub
- Think Like a CTO
- MongoDB 8.0 in Action, Third Edition
- Machine Learning System Design
What to know first
- data integrityData integrity is the property of data being accurate, complete, consistent, and valid throughout its storage, processing, and transfer.
- schemaA data schema is a formal specification of the structure, fields, types, relationships, and constraints of data.
- data governanceData governance is the organizational framework of policies, roles, responsibilities, processes, and controls that directs how data is managed, accessed, protected, and used.
- JSONJSON is a language-independent text-based data-interchange format that represents structured values using objects, arrays, strings, numbers, booleans, and null.
Related concepts
- schema registryused by · A schema registry is a service that stores, versions, and provides schemas for structured data, often validating compatibility between schema versions.
- Apache Icebergimplemented by · Apache Iceberg is an open table format for managing large analytical datasets with metadata, schema evolution, snapshots, and transactional operations across distributed or object storage.
- environment variableconfigured by · An environment variable is a named value supplied through a process's execution environment and made available to the process at runtime.
- source connectorused by · A source connector is a Kafka Connect connector that reads data from an external source system and publishes records to Kafka.
Questions about schema evolution
What is schema evolution?
Schema evolution is the controlled modification of a data schema over time while managing compatibility with existing data, producers, consumers, or stored records.
Which Manning books explain schema evolution?
Kafka for Architects, Architecting an Apache Iceberg Lakehouse, Kafka Streams in Action, Second Edition and Apache Kafka in Action.
What should I know before learning schema evolution?
Manning books that teach schema evolution usually introduce data integrity, schema, data governance and JSON first.
What is schema evolution also called?
Schema evolution is also called schema versioning or schema change.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



