Apache Parquet stores heterogeneous tabular data in a typed, column-oriented binary format. Unlike CSV, it can read only the columns needed and use compression and column encodings to reduce storage. Size still depends on the data and its encoding: a January 2020 New York City taxi dataset produced a 111 MB Parquet file, compared with a 105 MB compressed CSV and a 567 MB uncompressed CSV. section 8.2 section 8.5
Convert a CSV with PyArrow
Read the taxi CSV with `pyarrow.csv.read_csv`, then write its Arrow table with `pyarrow.parquet.write_table(table, "202001.parquet")`. This is the straightforward conversion when the table fits in memory. section 8.2
For parallel processing, a Parquet dataset can be partitioned into directories by column values. Partitioning the taxi data by `VendorID` and `passenger_count` creates paths for each combination, so separate files can be loaded or processed independently. Remove or otherwise handle null partition values before writing; do not discard rows merely for convenience. When reading an individual partition file, its partition columns may be absent because their values are encoded in the directory path. Inspect the files actually written rather than assuming one file per partition. section 8.2.3
When the source cannot fit in memory, pandas `read_csv(chunksize=1000000)` yields data frames one at a time. Convert each chunk to an Arrow table and pass it to a `ParquetWriter`; each write creates a row group in the same file. Specify stable column types before writing, because pandas may change an integer column to float in a chunk containing nulls. `ParquetFile` metadata reports the row groups, which can be read individually instead of loading the whole table. section 8.3.2
first = True
writer = None
for chunk in table_chunks:
chunk_table = pa.Table.from_pandas(chunk) #1
schema = chunk_table.schema
if first:
first = False
writer = pq.ParquetWriter(
"output.parquet", schema=schema) #2
writer.write_table(chunk_table)
writer.close()
The writer loop shows how successive pandas chunks become row groups in one Parquet file. section 8.3.2
Parquet also supports distributed persistence: Dask partitions can write separate files into one dataset directory, which PyArrow can read as a table. Python implementations do not expose every feature of the Parquet specification, including full control over column encoding or inspection of how a column is stored. section 10.2.5 section 8.2
