Partition pruning eliminates partitions or files that cannot satisfy a query filter before they are scanned. It reduces object-store listings, file reads, latency, and compute consumption. The architectural value depends on moving filtering into query planning rather than discovering irrelevant data late during execution. section 1.5 section 2.6.5
A metadata-driven pruning path typically works as follows:
- Use table and manifest-list statistics to exclude partitions that cannot match the filter. section 2.3
- Use manifest-file metadata and file-level statistics to exclude individual Parquet files before scanning their contents. section 2.3 section 3.4.3
- Use Parquet row-group statistics and predicate pushdown to skip nonmatching row groups within selected files. section 5.3.1
Partition pruning is most effective when storage serves files consistently and predictably. Compression, file-size tuning, caching, and parallel execution complement pruning, while variable storage performance can reduce its benefits. Teams should benchmark representative sequential reads, tail latency, and concurrency rather than relying only on advertised throughput. section 5.1.1 section 5.3.1
Filtering a sales table in Dremio
When a query searches for records whose sales_amount is 35, Dremio can skip a file whose metadata reports a sales_amount range of 38 to 40. Dremio’s native Iceberg integration also supports partition pruning and other Iceberg-aware performance enhancements, allowing teams to query lakehouse tables without duplicating data into a warehouse. section 3.4.3 section 8.5.2
Figure 2.4 Engines use metadata statistics to exclude data files from scans, speeding up queries.
Use the metadata-flow diagram to trace how statistics exclude irrelevant data files before scanning. section 2.3
