Repartitioning redistributes records across partitions, optionally changing their number or assigning index ranges to new partitions. In Dask, it can match partition granularity to changing workload intensity: use more partitions for expensive stages and fewer for lighter ones. The operation can also transmit data across nodes, so its benefit must outweigh redistribution cost. section 10.2.4 section 10.4
Apply repartitioning with these constraints:
- Changing only the partition count may combine two existing partitions into one uneven partition; inspect the task graph and consider balanced divisions. section 10.2.4
- Repartition by index divisions requires usable index metadata. Dask may expose delayed index computations or None-valued divisions, making optimization expensive or impossible. section 10.2.4
- Persisting data can avoid repeatedly repartitioning results, but persistence consumes memory and does not eliminate the cost of moving data when repartitioning is required. section 10.2.2 section 10.4
Reducing and balancing tax-data partitions
A Dask data frame named taxes can be reduced from three partitions to two with taxes.repartition(npartitions=2). To create explicit ranges, first set Geo_Name as the index, then repartition with divisions ["Alabama", "New Hampshire", "Wyoming"], producing two partitions with those boundaries. Visualizing the task graph helps reveal whether the resulting partitions are balanced. section 10.2.4
taxes2 = taxes.repartition(npartitions=2) taxes2.visualize(filename="10-repart.svg", rankdir="LR")
Inspect the runnable partition-count change and its task graph. section 10.2.4
