A dataset split divides data into parts that can be stored or processed independently. The split can follow column values, as in a partitioned Parquet dataset, or input size, as in a Dask data frame read in blocks. These choices affect which work can run concurrently and how much data must move between workers. section 8.2.3 section 10.2.1
Choose the split to match the work:
- For column-based storage, PyArrow can partition a Parquet dataset by VendorID and passenger_count. The output is a directory tree organized by those values, not one file. Remove null partition values only if doing so is valid for the actual task; dropping rows merely to simplify partitioning changes the dataset. section 8.2.3
- For bounded input processing, Dask’s read_csv(blocksize=5000) splits a small CSV into three partitions. Each Dask data-frame partition is implemented as a pandas data frame; task-graph visualization shows the resulting separate tasks. section 10.2.1
- For later stages, repartition when the useful task granularity changes. Reducing three Dask partitions to two joins two existing partitions and need not balance their sizes; index-based divisions offer another route but may require costly index computation or setup. section 10.2.4
from pyarrow import csv
import pyarrow.compute as pc
import pyarrow.parquet as pq
table = csv.read_csv(
"../../07-pandas/sec1-intro/yellow_tripdata_2020-01.csv.gz")
table = table.filter(
pc.invert(table["VendorID"].is_null())) #1
table = table.filter(pc.invert(table["passenger_count"].is_null()))
pq.write_to_dataset(
table, root_path="all.parquet",
partition_cols=["VendorID", "passenger_count"])
The PyArrow example filters null partition values, then writes a Parquet dataset under directories keyed by two columns. Its filtering step is an exercise-specific choice, not a general rule for preparing data. section 8.2.3
A split pays off when operations stay local. Loading separate Parquet files permits work by multiple processes or machines and can support concurrent writes. In Dask, an elementwise column calculation runs across partitions without communication. A maximum needs partition results gathered for a final calculation; sorting can require a more complex, expensive task graph. Inspect the graph rather than assuming every split makes an operation faster. section 8.2.3 section 10.2.3
Keep intermediate data partitioned when later queries need it: Dask’s .persist can retain computed partitions across the computing environment, whereas .compute brings the result back and can overwhelm local memory. Persistence also consumes memory, and moving data between workers costs time. For durable output, Dask can write one CSV per partition or a Parquet directory readable as one dataset; this disk storage is distinct from Dask’s .persist. section 10.2.2 section 10.2.5
Task count matters, too
For a 1000 × 1000 Dask array, one task per cell would create one million tasks. Rechunking into four blocks and applying a blockwise initialization creates four tasks instead. A useful split therefore considers task overhead as well as how much data each worker handles. section 10.3.2
