A row group is a storage and processing unit containing a subset of rows in a Parquet columnar file. It stores every column for those rows, while larger files may contain multiple row groups. section 8.2.1
Row groups support practical file inspection and chunked processing:
- Inspect metadata to count row groups, rows, columns, and total byte size. section 8.2.1
- Write successive tables with ParquetWriter; each table becomes a separate row group. section 8.3.2
- Read one row group at a time when memory or processing requirements make whole-file loading unsuitable. section 8.3.2
- Treat row-group sizing as a chunking choice, while recognizing that dataset partitioning may further split data into files or row groups. section 8.3.2 section 8.2.3
Write and read row groups
Read CSV data in one-million-row chunks, convert each chunk to a PyArrow table, write each table through ParquetWriter, then iterate over pf.read_row_group(groupi) instead of loading the complete file. section 8.3.2
parquet_file = pq.ParquetFile("202001.parquet")
metadata = parquet_file.metadata
print(metadata)
print(parquet_file.schema)
group = metadata.row_group(0)
print(group)
Inspect ParquetFile metadata, schema, and the first row group to see its row count, columns, and byte size. section 8.2.1
