Schema evolution is the controlled modification of a dataset schema over time while preserving compatibility with existing data and readers. In a lakehouse, it separates logical table structure from physical files, allowing long-lived datasets to adapt without routine reloads or rewrites. section 1.7 section 2.6.2
Apache Iceberg supports schema changes through table-level metadata and stable field identities:
- Add, rename, reorder, or delete columns without rewriting existing data files or breaking compatible queries. section 2.4 section 6.1.3
- Use stable field IDs to preserve a column's logical identity independently of its name or position, allowing older and newer schemas to coexist. section 6.1.3
- Track schema history at the table level for auditing and long-term data management. section 2.6.2
- Change partitioning separately through partition evolution, avoiding data migration when physical access strategies change. section 2.6.2
- Capture irregular attributes in variant columns when source structure changes faster than governed columns can safely evolve. section 6.1.3
Treat schema evolution as an ingestion and governance workflow rather than an automatic storage switch:
- Read the current table schema at the start of each batch or micro-batch job. section 6.1.3
- Inspect incoming fields and types, then validate them against the table definition. section 6.1.3
- Apply a deliberate response: extend the table, map the field into a variant column, or cast it to a compatible type. section 6.1.3
- Use schema registries or structured typing to track versions and flag incompatible changes before storage. section 6.5.4
- Monitor metadata churn, nested-field complexity, column counts, and snapshot growth; use compaction or snapshot pruning when required. section 6.5.4 section 6.5.2
Evolving mobile application events
A product analytics platform receives events from several mobile applications. New fields appear as the applications change, so ingestion uses schema-registry support to track compatible versions, writes in append or merge-on-read mode, and alerts on mismatches before they reach the Iceberg table. This preserves downstream access while making incompatible changes visible. section 6.5.4
Table 4.3 Findings from interviews and audit
| Domain |
Key findings |
Problems to address |
|---|---|---|
| Storage |
Fragmented infrastructure across HDFS, NAS, and cloud warehouses |
Data duplication, inconsistent governance, and an unclear long-term storage strategy |
| Processing |
Mix of Spark, Kafka, NiFi, and Informatica across batch and streaming jobs |
Tool incompatibility, redundant pipelines, and manual schema coordination |
| Governance |
Metadata scattered across Hive, relational DBs, and manual processes |
Manual schema evolution, audit complexity, and inconsistent access controls |
| Data consumption |
Use of Dremio, Presto, Snowflake, Tableau, and Power BI |
Semantic drift across tools, multiple data copies, and inconsistent user experiences |
Use the audit table to compare manual schema coordination with the governance, compatibility, and access-control problems that controlled evolution addresses. section 4.2.3
