Apache Arrow is a language-independent, in-memory format for columnar data, supported by libraries for reading, processing, and sharing that data. PyArrow exposes lower-level implementations to Python. A practical role for Arrow is to accelerate selected parts of a pandas workflow, not to replace pandas analysis wholesale. section 7.4.1
Start with data loading. Arrow’s CSV reader uses multiple threads and infers column types. On a January 2020 New York City taxi CSV, loading took about 2 seconds versus 12 seconds with pandas on the measured machine. The resulting Arrow table occupied 865 MB versus 2 GB for pandas without type guidance; a manually optimized pandas load used 250 MB. Those figures make the comparison useful, but not a claim that Arrow always uses less memory. section 7.4.2
from pyarrow import csv #1
table = csv.read_csv("../sec1-intro/yellow_tripdata_2020-01.csv.gz")
tot_bytes = 0
for name in table.column_names: #2
col_bytes = table[name].nbytes
col_type = table[name].type
print(name, col_bytes // (1024 ** 2))
tot_bytes += col_bytes
print("Total", tot_bytes // (1024 ** 2))
Read the CSV with pyarrow.csv, then inspect each column’s type and byte count before deciding whether its inferred representation needs adjustment. section 7.4.2
Arrow types are not identical to pandas or NumPy types, even though conversion is straightforward. Arrow records missing values in a separate bit array, so an integer column can contain nulls without using a null value as an integer. Domain knowledge can still improve inferred types: a column with only a few values may need a smaller representation, while Arrow cannot assume reduced floating-point precision is acceptable. Convert a loaded table for pandas work with `table.to_pandas()`. section 7.4.2
Compute without first converting to pandas
For taxi tips, filter rows whose total is nonzero, divide `tip_amount` by `total_amount`, and take the mean with `pyarrow.compute`. The measured calculation took about 15 ms, roughly half the equivalent pandas time on that machine. Arrow’s compute interface is lower-level and type-sensitive: the illustrated comparison requires `0.0`, not integer `0`. Its analysis features remain more limited than pandas’. section 7.4.3
The shared memory format also changes the cost of interoperability. Passing data already represented as Arrow between implementations can amount to sharing a memory pointer. Starting with a pandas frame and ending with an R frame still requires conversions, with associated time and potentially additional memory. For separate local processes, the illustrated Plasma server stores Arrow objects in shared memory and lets producers and consumers find them through a catalog; it is a local-socket example, not a network service. section 7.5.1 section 7.5.2
Arrow also connects in-memory processing to persistence. PyArrow can read a CSV table and write it as Parquet. On a memory-constrained machine, pandas can supply CSV chunks that become Arrow tables written successively as Parquet row groups; keeping column types consistent across chunks matters when nulls change pandas’ inferred types. section 8.2 section 8.3.2
