Metadata design organizes the information that lets a lakehouse locate, interpret, version, and plan access to table data. Apache Iceberg replaces broad file and directory inspection with a structured, multilayered system that guides query engines toward relevant files. section 2.3
An Iceberg table separates responsibilities across metadata layers:
- Table-level metadata tracks snapshots, schema definitions, and partitioning strategies. section 2.3
- A catalog provides a stable entry point for locating table definitions and their metadata layers. section 2.3
- Metadata enables data pruning during query planning, reducing unnecessary file scans before execution. section 2.3
- Snapshots support time travel, auditing, debugging, and rollback without maintaining separate table copies. section 1.6
The design also supports schema evolution, partition evolution, and hidden partitioning. Partition strategies can change as datasets grow, while transform-based partition logic lets engines prune efficiently without requiring exposed day or month columns in queries. These capabilities improve flexibility and operational scalability, although implementations still differ across Iceberg, Hudi, Delta Lake, Paimon, and interoperability layers. section 1.6 section 2.3
Figure 2.3 The structure and flow of an Apache Iceberg table read-and-write operation
Use the read-and-write flow to trace how catalogs, layered metadata, and data files cooperate during table access. section 2.3
