Parquet
Apache Parquet is an open, column-oriented binary file format for storing and processing structured data with schema and metadata support.
Architecting an Apache Iceberg Lakehouse
Apache Parquet is an open columnar file format designed for efficient storage and analytical processing of large datasets.
- How it is used
- Supports efficient analytical storage as the primary physical file format beneath Iceberg tables.
- How deep it goes
- Explains column layout, row groups, compression, statistics, pruning, and execution, without descending into the complete file specification.
- Examples
- Uses Spark, MinIO, Iceberg snapshots, row-group footers, and ingestion tools that convert or write Parquet data.
- What you can do after
- Evaluate storage architectures by relating Parquet’s pruning, compression, interoperability, and query-performance benefits to operational requirements.
Read how Architecting an Apache Iceberg Lakehouse explains Parquet →
DuckDB in Action
Parquet is a column-oriented file format for storing structured data together with schema and related metadata.
- How it is used
- Supports direct analytical querying, format conversion, data-lake storage, and large-file export without requiring database persistence.
- How deep it goes
- Examines schemas, physical types, row-group statistics, compression, and pushdown, rather than treating Parquet as an opaque file.
- Examples
- Uses ATP rankings, Stack Overflow tables, and monthly New York taxi files to demonstrate inspection, export, and large-scale querying.
- What you can do after
- Apply it via DuckDB to inspect, query, export, compress, and benchmark structured datasets stored as files or cloud objects.
The Well-Grounded Data Analyst
Parquet is a compressed columnar file format that preserves data types and is designed for efficient analytical storage and retrieval.
- How it is used
- Separates data cleaning from repeated analytical work by storing a reusable intermediate dataset.
- How deep it goes
- Defines compression and type preservation, then demonstrates export; it does not explain the internal file structure.
- Examples
- Exports cleaned event data to a gzip-compressed Parquet file after correcting product categories, brands, and names.
- What you can do after
- Operate Parquet through pandas to save cleaned analytical data and reload typed columns for later analysis.
Fast Python: High performance techniques for large datasets
- How it is used
- Stores typed tabular data for selective column access and partition-based processing.
- How deep it goes
- Works through CSV conversion and directory partitioning, while noting that Python cannot fully control or inspect column encodings.
- Examples
- Converts January 2020 New York City taxi trips from CSV and partitions them by vendor and passenger count.
- What you can do after
- Apply it via PyArrow and Dask to write, partition, and read datasets without collecting every distributed partition in memory.
Read how Fast Python: High performance techniques for large datasets explains Parquet →
Machine Learning Platform Engineering
Apache Parquet is a column-oriented, binary file format designed for efficient storage and retrieval of structured data.
- How it is used
- Uses compressed Parquet artifacts to move larger tabular datasets through reproducible ML preprocessing pipelines.
- How deep it goes
- Demonstrates conversion, compression, splitting, object-storage reads, and QA without detailing Parquet’s internal encoding.
- Examples
- Uses MovieLens 25M ratings and movie tables, compressed output files, MinIO objects, and a PyArrow training-file QA check.
- What you can do after
- Build it from scratch as a Kubeflow data-preparation pipeline that converts, splits, stores, and validates Parquet datasets.
Read how Machine Learning Platform Engineering explains Parquet →
Designing Deep Learning Systems: A guide for software engineers
Apache Parquet is an open column-oriented file format designed for efficient storage and retrieval of structured data.
- How it is used
- Supports efficient structured-data storage and framework-neutral exchange in deep-learning dataset pipelines.
- How deep it goes
- Defines the format and shows its conversion role, but stops short of explaining Parquet internals.
- Examples
- Uses a flower-image ETL pipeline and Petastorm conversion into a PyTorch training dataset.
- What you can do after
- Apply Petastorm to move Delta Lake data into TensorFlow or PyTorch datasets without framework-specific conversion details.
Read how Designing Deep Learning Systems: A guide for software engineers explains Parquet →
Digital Twins in Action
Parquet is a column-oriented, open-source file format for efficient storage and retrieval of structured data.
- How it is used
- Stores historical analytical data and prepared machine-learning features outside transactional serving stores.
- How deep it goes
- Explains selective column reads, compression, schema metadata, and immutable-file limits without detailing encoding internals.
- Examples
- Uses a DuckDB query over taxi trips and a sensor-data pipeline that produces resampled Parquet features.
- What you can do after
- Apply it via DuckDB to query a Parquet file directly and use Parquet to store prepared machine-learning features.
Machine Learning System Design
Apache Parquet is a columnar storage file format designed for efficient storage and retrieval of structured data.
- How it is used
- Uses Parquet to store production data separately for simpler reads during dataset gathering and preparation.
- How deep it goes
- Treats Parquet as a practical storage choice and stops short of explaining its format internals or advanced preprocessing.
- Examples
- Uses daily PhotoStock search and purchase data, stored as Parquet before query–image labeling.
- What you can do after
- Operate/configure daily batch extraction that writes production data to Parquet for downstream dataset work.
Financial AI in Practice
Parquet is a column-oriented file format for storing structured data with efficient compression and schema metadata.
- How it is used
- Persists standardized article-fetcher outputs in a scalable news-data pipeline.
- How deep it goes
- Compares practical storage benefits without detailing Parquet’s internal encoding or file structure.
- Examples
- Saves fetched news articles from NewsAPI and RSS sources as Parquet files.
- What you can do after
- Apply it via pandas to persist standardized fetcher outputs as Parquet files.
Related concepts
- CSValternative to · CSV is a plain-text format for tabular data, representing records as rows and fields as values separated by delimiters, conventionally commas, with quoting conventions for special content.
- columnar datakind of · Columnar data is tabular data organized and stored by column rather than by complete row, enabling efficient access to selected fields and data compression.
- compressionuses · Compression is the transformation of information into a more compact representation to reduce storage or transmission requirements while retaining information needed for recovery or use.
- pipelineproduced by · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
- DuckDBused by · DuckDB is an open-source, in-process analytical database management system with a columnar-vectorized execution engine for fast local querying.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.