A Dask dataframe provides a pandas-like interface while representing operations as a task graph instead of immediately producing results. The graph records operation nodes and dependency edges; computation begins only when explicitly triggered. section 10.1.2
Use Dask dataframes with these execution and performance distinctions in mind:
- Expect many familiar pandas operations, but check compatibility because features can be missing or behave differently. section 10.1.2
- Exploit partition-local transformations, which can run in parallel without communication between partitions. section 10.2.3
- Inspect task graphs before expensive reductions or sorting; aggregation, communication, barriers, and shuffles can bottleneck parallelism. section 10.2.3
Computing property-tax fractions
Read a tab-separated tax dataset, clean and convert its Amount column, mark Tax_Type as a known categorical column, and build a pivot table. Selecting the property-tax fraction constructs a graph; visualize it to inspect dependencies, then call compute() to return a pandas dataframe. A column conversion can remain partition-parallel, while max() requires partition results to be transferred and combined by a single aggregation task. section 10.1.2 section 10.2.3
import numpy as np
import dask.dataframe as dd #1
taxes = dd.read_csv("FY2016-STC-Category-Table.csv", sep="\t")
taxes["Amount"] = taxes["Amount"].str.replace(",",
"").replace("X", np.nan).astype(float)
taxes["Tax_Type"] = taxes["Tax_Type"].astype(
"category").cat.as_known() #2
pivot = taxes.pivot_table(index="Geo_Name",
columns="Tax_Type", values="Amount")
has_property_info = pivot[
~pivot["Property Taxes"].isna()].index #3
pivot_clean = pivot.loc[has_property_info]
frac_property = pivot_clean["Property Taxes"] / pivot_clean["Total Taxes"]
Runnable code for constructing the tax dataframe, cleaning Amount, creating the pivot, and deriving the property-tax fraction. section 10.1.2
