Parquet
Apache Parquet is an open, column-oriented binary file format for storing and processing structured data with schema and metadata support.
Architecting an Apache Iceberg Lakehouse
Apache Parquet is an open columnar file format designed for efficient storage and analytical processing of large datasets.
- How it is used
- Supports efficient analytical storage as the primary physical file format beneath Iceberg tables.
- How deep it goes
- Explains column layout, row groups, compression, statistics, pruning, and execution, without descending into the complete file specification.
- Examples
- Uses Spark, MinIO, Iceberg snapshots, row-group footers, and ingestion tools that convert or write Parquet data.
- What you can do after
- Evaluate storage architectures by relating Parquet’s pruning, compression, interoperability, and query-performance benefits to operational requirements.
Read how Architecting an Apache Iceberg Lakehouse explains Parquet →
DuckDB in Action
Parquet is a column-oriented file format for storing structured data together with schema and related metadata.
- How it is used
- Supports direct analytical querying, format conversion, data-lake storage, and large-file export without requiring database persistence.
- How deep it goes
- Examines schemas, physical types, row-group statistics, compression, and pushdown, rather than treating Parquet as an opaque file.
- Examples
- Uses ATP rankings, Stack Overflow tables, and monthly New York taxi files to demonstrate inspection, export, and large-scale querying.
- What you can do after
- Apply it via DuckDB to inspect, query, export, compress, and benchmark structured datasets stored as files or cloud objects.
The Well-Grounded Data Analyst
Parquet is a compressed columnar file format that preserves data types and is designed for efficient analytical storage and retrieval.
- How it is used
- Separates data cleaning from repeated analytical work by storing a reusable intermediate dataset.
- How deep it goes
- Defines compression and type preservation, then demonstrates export; it does not explain the internal file structure.
- Examples
- Exports cleaned event data to a gzip-compressed Parquet file after correcting product categories, brands, and names.
- What you can do after
- Operate Parquet through pandas to save cleaned analytical data and reload typed columns for later analysis.
Fast Python: High performance techniques for large datasets
- How it is used
- Stores heterogeneous columnar datasets efficiently for selective reads, larger-than-memory processing, and distributed writes.
- How deep it goes
- Works through conversion, partitioning, and row-group processing while noting incomplete Python access to encoding controls and inspection.
- Examples
- Uses New York City taxi CSV data for file-size comparison, partitioning, and chunked conversion.
- What you can do after
- Apply it via PyArrow to write, partition, and read Parquet datasets and row groups.
Read how Fast Python: High performance techniques for large datasets explains Parquet →
Machine Learning Platform Engineering
Apache Parquet is a column-oriented, binary file format designed for efficient storage and retrieval of structured data.
- How it is used
- Uses compressed Parquet artifacts to move larger tabular datasets through reproducible ML preprocessing pipelines.
- How deep it goes
- Demonstrates conversion, compression, splitting, object-storage reads, and QA without detailing Parquet’s internal encoding.
- Examples
- Uses MovieLens 25M ratings and movie tables, compressed output files, MinIO objects, and a PyArrow training-file QA check.
- What you can do after
- Build it from scratch as a Kubeflow data-preparation pipeline that converts, splits, stores, and validates Parquet datasets.
Read how Machine Learning Platform Engineering explains Parquet →
Designing Deep Learning Systems: A guide for software engineers
Apache Parquet is an open column-oriented file format designed for efficient storage and retrieval of structured data.
- How it is used
- Supports efficient structured-data storage and framework-neutral exchange in deep-learning dataset pipelines.
- How deep it goes
- Defines the format and shows its conversion role, but stops short of explaining Parquet internals.
- Examples
- Uses a flower-image ETL pipeline and Petastorm conversion into a PyTorch training dataset.
- What you can do after
- Apply Petastorm to move Delta Lake data into TensorFlow or PyTorch datasets without framework-specific conversion details.
Read how Designing Deep Learning Systems: A guide for software engineers explains Parquet →
Digital Twins in Action
Parquet is a column-oriented, open-source file format for efficient storage and retrieval of structured data.
- How it is used
- Stores portable columnar data for historical analysis and machine-learning pipelines.
- How deep it goes
- Explains selective column reads, compression, footer metadata, and file immutability, with a direct-query example.
- Examples
- Uses taxi fare aggregation and a home sensor-data pipeline to illustrate analysis and feature storage.
- What you can do after
- Apply it via DuckDB to query a Parquet file with SQL without first loading it into a database.
Machine Learning System Design
Apache Parquet is a columnar storage file format designed for efficient storage and retrieval of structured data.
- How it is used
- Uses Parquet to store production data separately for simpler reads during dataset gathering and preparation.
- How deep it goes
- Treats Parquet as a practical storage choice and stops short of explaining its format internals or advanced preprocessing.
- Examples
- Uses daily PhotoStock search and purchase data, stored as Parquet before query–image labeling.
- What you can do after
- Operate/configure daily batch extraction that writes production data to Parquet for downstream dataset work.
Financial AI in Practice
Parquet is a column-oriented file format for storing structured data with efficient compression and schema metadata.
- How it is used
- Persists standardized article-fetcher outputs in a scalable news-data pipeline.
- How deep it goes
- Compares practical storage benefits without detailing Parquet’s internal encoding or file structure.
- Examples
- Saves fetched news articles from NewsAPI and RSS sources as Parquet files.
- What you can do after
- Apply it via pandas to persist standardized fetcher outputs as Parquet files.
Also covered in
- Software Engineering for Data ScientistsIn early access (MEAP)
- Cloud Observability in Action
- Hugging Face in Action
What to know first
- CSVCSV is a text format for tabular data in which records contain fields separated by commas, with conventions for quoting fields that contain special characters.
- data frameA data frame is a two-dimensional tabular data structure organized into labeled rows and columns, often with columns that can contain different types of values.
- columnar storageColumnar storage is a data layout that stores values grouped by column rather than by row.
- JSONJSON is a language-independent text-based data-interchange format that represents structured values using objects, arrays, strings, numbers, booleans, and null.
Related concepts
- CSValternative to · CSV is a text format for tabular data in which records contain fields separated by commas, with conventions for quoting fields that contain special characters.
- columnar storagekind of · Columnar storage is a data layout that stores values grouped by column rather than by row.
- compressionuses · Compression is the encoding of data or information in a more compact representation to reduce its size while retaining content needed for recovery or use.
- DuckDBused by · DuckDB is an open-source, in-process analytical database management system with a columnar-vectorized execution engine for fast local querying.
- pipelineproduced by · A pipeline is an ordered sequence of connected processing stages through which data or work flows, with each stage passing or transforming its output for subsequent stages.
Questions about Parquet
What is Parquet?
Apache Parquet is an open, column-oriented binary file format for storing and processing structured data with schema and metadata support.
Which Manning books explain Parquet?
Architecting an Apache Iceberg Lakehouse, DuckDB in Action, The Well-Grounded Data Analyst, Fast Python: High performance techniques for large datasets, Machine Learning Platform Engineering, Designing Deep Learning Systems: A guide for software engineers, Digital Twins in Action, Machine Learning System Design and 1 more.
What should I know before learning Parquet?
Manning books that teach Parquet usually introduce CSV, data frame, columnar storage and JSON first.
What is the difference between Parquet and CSV?
Apache Parquet is an open, column-oriented binary file format for storing and processing structured data with schema and metadata support. CSV is a text format for tabular data in which records contain fields separated by commas, with conventions for quoting fields that contain special characters.
What is Parquet a kind of?
Parquet is a kind of columnar file format. A columnar file format stores values grouped by column rather than complete records grouped by row, supporting efficient analytical access.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.








