liveBook · Manning

Parquet

As explained in Machine Learning System Design

Apache Parquet is a columnar storage file format designed for efficient storage and retrieval of structured data.

When search and purchase data grows large, storing extracted production data separately as Parquet files provides a simpler read path for dataset preparation. Full datasets can remain available for processing, while subsampling is reserved for correcting imbalanced labels. section 6.7.2

PhotoStock dataset pipeline

A daily batch job orchestrated with Flyte gathers data from production databases and writes it to Parquet files. The resulting dataset supports labeling query–image pairs as Relevant, Irrelevant, or Can't answer without requiring sophisticated preprocessing. section 6.7.2

How Machine Learning System Design covers it

How deep it goes
Treats Parquet as a practical storage choice and stops short of explaining its format internals or advanced preprocessing.
Examples
Uses daily PhotoStock search and purchase data, stored as Parquet before query–image labeling.
What it assumes
Assumes production databases and batch orchestration, while leaving sophisticated preprocessing unnecessary for the initial workflow.
What you can do after
Operate/configure daily batch extraction that writes production data to Parquet for downstream dataset work.
How it connects
Connects Parquet with ETL, batch processing, Flyte, production databases, labeling, and class balancing.
Cover of Machine Learning System Design
liveBookmanning.com
Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

Compare how 9 books explain Parquet →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.