Row group
A row group is a storage and processing unit containing a subset of rows in a columnar data file.
DuckDB in Action
A row group is a horizontal partition of rows in a Parquet file or DuckDB storage format that is stored and processed as a unit.
- How it is used
- Organizes columnar data for compression, parallel reads, and metadata-driven query skipping.
- How deep it goes
- Explains partitioning, sizing trade-offs, metadata statistics, and runtime processing, but stops short of storage-page internals.
- Examples
- Demonstrates ATP rankings conversion, a 100,000-row Parquet setting, metadata inspection, and date-based row-group skipping.
- What you can do after
- Operate DuckDB to configure Parquet row-group sizes and inspect row-group metadata for query-performance decisions.
Architecting an Apache Iceberg Lakehouse
A row group is a storage and I/O unit containing a subset of a columnar file's rows.
- How it is used
- Supports selective reads and predicate-based skipping within Parquet files used by Iceberg lakehouses.
- How deep it goes
- Explains storage layout, column chunks, footer statistics, and sizing tradeoffs, but not row-group creation internals.
- Examples
- Uses Parquet footer metadata diagrams and compaction commands targeting 256 MB output files.
- What you can do after
- Diagnose memory, spill, parallelism, and scan-efficiency issues by evaluating row-group and file-size choices.
Read how Architecting an Apache Iceberg Lakehouse explains row group →
Fast Python: High performance techniques for large datasets
- How it is used
- Organizes Parquet data for metadata inspection, chunked writing, and memory-aware row-by-row-group processing.
- How deep it goes
- Defines row-group contents and demonstrates metadata, writing, and reading, but does not descend into lower-level storage internals.
- Examples
- Uses a 6,405,008-row file with one row group, seven generated row groups, and one-million-row input chunks.
- What you can do after
- Operate/configure PyArrow ParquetWriter and ParquetFile to create, inspect, and process row groups incrementally.
Read how Fast Python: High performance techniques for large datasets explains row group →
Related concepts
- query pushdownused by · Query pushdown is an optimization that evaluates eligible filtering, projection, or aggregation operations as close as possible to the source data.
- dataset splitkind of · A dataset split is the division of a dataset into distinct subsets assigned different purposes, such as training, validation, or testing.
- Parquetused by · Apache Parquet is an open, column-oriented binary file format for storing and processing structured data with schema and metadata support.
- Apache Icebergpart of · Apache Iceberg is an open table format for managing large analytical datasets with metadata, schema evolution, snapshots, and transactional operations across distributed or object storage.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.