Table metadata is the structured state that lets query engines understand an Iceberg table without searching every data file. It describes schemas, partitioning, versions, and file relationships, enabling planning-time pruning, schema evolution, time travel, and consistent access at scale. section 2.3
Iceberg distributes this description across complementary metadata layers:
- Table-level metadata in metadata.json records snapshots, schema definitions, and partitioning strategies. section 2.3
- Snapshot-level manifest lists represent one table state and summarize groups of manifests for scan reduction. section 2.3
- File-level manifests list individual Parquet files, locations, and statistics used to exclude irrelevant files. section 2.3
A catalog provides the shared entry point and authoritative association between a table identifier and its current metadata. It registers tables, organizes namespaces, updates metadata locations transactionally, and coordinates optimistic commits across engines. section 7.1.1
Pruning and time travel
In Dremio, per-file statistics can skip a file whose sales_amount range is 38–40 when searching for 35. Metadata-table queries expose snapshot IDs, which can then select an earlier table state or timestamp for auditing and error investigation. section 3.4.3
Figure 2.3 The structure and flow of an Apache Iceberg table read-and-write operation
Shows how engines and storage interact with Iceberg metadata during table reads and writes. section 2.3
