Parquet is an open-source, column-oriented file format for structured data. It suits analytical work that filters, aggregates, or reads selected fields across many records. Unlike a database, a Parquet file can sit in a filesystem or object store and be queried directly by a compatible engine. section 4.3.2
Consider sensor readings with timestamps, temperature, humidity, and pressure. A column-oriented layout groups values by field rather than keeping every reading together. A query that needs only temperature can avoid reading the other columns. Compression reduces file size, while schema and metadata in the file footer preserve data types. section 4.3.2 section 8.1.4
Query a file without loading it into a database
DuckDB can run SQL against a local Parquet file: SELECT COUNT(*) AS trips, ROUND(AVG(fare_amount), 2) AS avg_fare FROM 'taxi.parquet'. The query counts taxi trips and calculates their average fare directly from the file. section 4.3.2
Listing 4.4 Using DuckDB and SQL to query data directly from a Parquet file
import duckdb, requests, time
url = "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_
2024-01.parquet"
with open('taxi.parquet', 'wb') as f: #1
f.write(requests.get(url).content)
start_time = time.time()
result = duckdb.query(""" #2
SELECT
COUNT(*) as trips,
ROUND(AVG(fare_amount), 2) as avg_fare,
ROUND(MIN(fare_amount), 2) as min_fare,
ROUND(MAX(fare_amount), 2) as max_fare
FROM 'taxi.parquet'
""").fetchone()
query_time = time.time() - start_time
print(f"\nResults: {result[0]:,} trips, avg ${result[1]},
range ${result[2]}-${result[3]}")
The full listing downloads the taxi file, queries its trip count and fare statistics with DuckDB, and times the query. section 4.3.2
Parquet also serves as an output for analytical pipelines: data from multiple sources can be aligned, resampled, cleaned, and stored as features for machine learning. It is not, by itself, a transactional table format. Individual files are immutable; changing a record or adding a column requires writing a new file. Delta Lake adds a transaction log and metadata layer over Parquet for record-level changes. section 8.1 section 4.3.5
