Metadata is descriptive information about data’s structure, properties, or content. In analytical workflows, file metadata can answer questions without scanning the underlying records. section 10.4.2
Parquet metadata can include:
- Row count. section 10.4.2
- Schema, including column names and types. section 10.4.2
- Minimum values, maximum values, and nullability for each column. section 10.4.2
Count rows without scanning the file
SELECT count(*) FROM read_parquet('users.parquet'); DuckDB can use Parquet metadata to return the row count quickly. The same operation on CSV requires reading the file, making Parquet nearly 1,000 times faster in the demonstrated comparison. section 10.3
SELECT count(*) FROM 's3://us-prd-md-duckdb-in-action/nyc-taxis/yellow_tripdata_2022-06.parquet'; -- 3,558,124 rows in 600 ms
Inspect a metadata-powered count against a large Parquet file stored remotely. section 10.4.2
