Metadata is structured information that describes data and helps systems locate, interpret, version, and manage it. In an Apache Iceberg lakehouse, metadata adds a standard layer over files such as Apache Parquet, allowing distributed files to behave as coherent relational tables on object storage. This improves reliability and query efficiency without changing the underlying file format. section 2.0 section 2.3
Iceberg uses layered metadata rather than one monolithic structure. Table-level metadata records schemas, partition specifications, and snapshot history; manifest lists and manifests connect snapshots to data files while carrying partition summaries, column statistics, and file paths. Query engines use these layers during planning to exclude irrelevant files before execution. section 2.3 section 10.1.3
Figure 2.3 The structure and flow of an Apache Iceberg table read-and-write operation
Trace how catalogs, table metadata, manifests, and data files cooperate during Iceberg table reads and writes. section 2.3
The metadata system supports several operational responsibilities:
- Tracks immutable table versions so readers can obtain consistent views, use time travel, and roll back to earlier states. section 7.1
- Provides catalogs with a stable entry point for resolving table names, locating current metadata, organizing namespaces, and coordinating updates across compute engines. section 7.1
- Exposes file, snapshot, manifest, partition, and size information through metadata views that can drive maintenance decisions. section 11.1.2
Inspecting an Iceberg table in MinIO
After Spark ingests data into an Iceberg table, open the MinIO warehouse bucket and navigate to the table location. Parquet files contain the data, while JSON metadata files and Avro manifest files describe the table structure and file organization. This inspection verifies that storage contains both the analytical data and the metadata needed to read it efficiently. section 3.3.5
Figure 2.4 Engines use metadata statistics to exclude data files from scans, speeding up queries.
Visualize how manifest statistics let query engines exclude data files from scans and reduce query-planning and I/O costs. section 2.3
Metadata also creates maintenance signals. High-frequency ingestion can produce many small manifests and an increasingly long snapshot history, increasing planning and commit costs. A scheduled metadata query can count active files below a target size, such as 20 MB, and trigger compaction only when fragmentation exceeds a defined limit. section 10.1.3 section 11.1.2
