File-level metadata is the Iceberg layer that describes individual data and delete files, including their paths, partitions, and statistics. Query engines use it to exclude irrelevant files during planning instead of inspecting the entire dataset. section 2.3 section 11.1.2
Its implementation supports several architectural and operational tasks:
- Connects manifests to individual Parquet files within a dataset. section 2.3
- Stores file paths, partitions, and statistics so engines can prune data before scanning. section 2.3
- Exposes active data and delete files through queryable metadata views backed by the Iceberg SDK. section 11.1.2
- Enables maintenance decisions such as triggering compaction when too many files fall below a target size, such as 20 MB. section 11.1.2
Figure 2.4 Engines use metadata statistics to exclude data files from scans, speeding up queries.
Shows how file statistics exclude data files from scans and improve query speed. section 2.3
