Dask runs parallel analytics on partitioned data, from one machine to a cluster. Its data-frame and array interfaces resemble pandas and NumPy, but can represent larger-than-memory objects. Similar-looking code has different execution semantics: most Dask operations build a task graph rather than immediately producing a result. Call `.compute()` when you need the result. section 10.0 section 10.1.2
From a data-frame expression to a result
Read a tab-separated tax file with `dd.read_csv`, clean its `Amount` column, and calculate property taxes divided by total taxes. Printing the resulting Dask object does not print the calculated fractions. Use `frac_property.visualize(...)` to inspect the planned operations, then `frac_property.compute()` to execute them and obtain a pandas result. Do not assume a pandas operation is interchangeable: some data-frame features are absent or behave differently. section 10.1.2
import numpy as np
import dask.dataframe as dd #1
taxes = dd.read_csv("FY2016-STC-Category-Table.csv", sep="\t")
taxes["Amount"] = taxes["Amount"].str.replace(",",
"").replace("X", np.nan).astype(float)
taxes["Tax_Type"] = taxes["Tax_Type"].astype(
"category").cat.as_known() #2
pivot = taxes.pivot_table(index="Geo_Name",
columns="Tax_Type", values="Amount")
has_property_info = pivot[
~pivot["Property Taxes"].isna()].index #3
pivot_clean = pivot.loc[has_property_info]
frac_property = pivot_clean["Property Taxes"] / pivot_clean["Total Taxes"]
The tax calculation shows familiar data-frame operations assembled into a Dask computation before execution. section 10.1.2
Partition size shapes the work. A CSV read with `blocksize=5000` splits a roughly 15 KB input into three partitions, each represented as a pandas data frame; Dask array partitions similarly use NumPy arrays. Independent arithmetic can run across partitions without communication. A global maximum instead combines partition maxima, reducing parallelism at the final step. Sorting can require still more complex coordination. Visualize an operation’s graph before assuming its distributed cost resembles its pandas cost. section 10.2.1 section 10.2.3
The `dask.distributed` scheduler assigns tasks to workers; a client submits work and dashboards expose execution and worker state. Worker processes, threads, and memory limits are configurable, but useful settings depend on the workload. Connected data-frame and array operations use the distributed scheduler. Keep work coarse enough to justify scheduling: initializing a 1,000 × 1,000 image one cell at a time would create one million tasks, whereas blockwise initialization works on array partitions. section 10.3.1 section 10.3.2
Watch memory and data movement as closely as CPU use. `.persist()` computes and retains partition results for reuse; it does not mean writing durable output. Calling `.compute().to_csv(...)` first gathers a data frame into pandas, which may not fit in memory. Distributed CSV output writes one file per partition, while Parquet supports partition-written output under one directory. Persisting intermediates can backfire when workers spill to disk: in one 10,000 × 10,000 Mandelbrot run, the persisted case took 752 seconds versus 696 without those persists. section 10.2.5 section 10.3.3
