Apache Parquet is a columnar file format designed for large-scale analytical workloads. It stores values by column rather than complete records by row, making it well suited to queries that scan many rows but select relatively few columns. In an Iceberg lakehouse, Parquet supplies the physical data files while Iceberg manages schemas, snapshots, metadata, and table semantics. section 2.1 section 5.3.1
Parquet improves analytical storage and query execution through several related mechanisms:
- Organizes files into row groups, with column chunks inside each group, so engines can read only requested columns. section 5.3.1
- Stores column-level statistics in row-group footers, allowing predicate pushdown to skip irrelevant row groups. section 5.3.1
- Supports compression techniques such as dictionary and run-length encoding, which benefit from repeated values within columns. section 5.3.1
- Enables vectorized execution by processing batches of values, improving CPU cache locality and use of SIMD instructions. section 2.1
Parquet’s advantages are architectural as well as local to each file. Its efficient I/O and compression reduce storage and scan costs, while its open adoption supports interoperability across engines and storage platforms. Iceberg adds table-level metrics derived from Parquet footers and registers newly written files in snapshot metadata, combining efficient file access with consistent table management. section 5.3 section 5.3.1
Parquet in an Iceberg lakehouse
A Spark ingestion job writes sales data as Parquet files into MinIO. Iceberg records those files in its metadata, alongside JSON metadata and Avro manifest files. Query engines can then use Parquet statistics and Iceberg metadata to locate and scan only relevant data. section 3.3.5 section 5.3.1
Figure 5.8 Using metadata in Parquet row-group footers, we can tell which row groups to scan and which to skip for faster reads.
Use the row-group footer diagram to trace how column statistics identify data to scan and data to skip. section 5.3.1
