A computation graph records work before Dask executes it. Its nodes are operations, and its edges are dependencies between operations. Building a Dask data-frame expression prepares this task graph; unlike an eager pandas or NumPy expression, it need not produce a result immediately. Calling compute() executes the graph and returns the result. section 10.1.2
Inspect a data-frame calculation
A property-tax calculation reads a CSV, cleans and converts an Amount column, constructs a pivot table, and divides property taxes by total taxes. Before computing the final fraction, frac_property.visualize(filename="10-property.svg", rankdir="LR") exports its task graph. The initial nodes show CSV reading followed by Amount conversion and assignment: the expression describes a sequence of dependent tasks, not an already calculated fraction. section 10.1.2
Graph shape exposes costs that similar-looking data-frame expressions can hide:
- Partitioning a CSV read into 5,000-byte blocks produces separate read-and-transform paths. In this example, a roughly 15 KB input becomes three partitions, each represented internally by a pandas data frame. section 10.2.1
- Dividing Amount by 1,000 runs independently across partitions. Finding the maximum first computes a maximum per partition, then combines those values at one node, reducing parallelism and requiring data transfer. section 10.2.3
- Sorting can produce a more complex graph with barrier and shuffle tasks. Visualizing the operation helps reveal communication and synchronization before choosing an approach. section 10.2.3
Persisting an intermediate result changes what later graphs must repeat. After parsing Amount into numbers, taxes = taxes.persist() makes that prepared state reusable while keeping its partitions distributed. Persist is useful for frequently reused, relatively small results, but retaining too much data consumes memory. By contrast, compute() brings the result back; for a data frame larger than local memory, that can fail. section 10.2.2
Choose tasks at the right granularity
Initializing a Dask array one pixel at a time creates nine tasks for a 3 × 3 Mandelbrot image—and would create one million for a 1000 × 1000 image. Processing independent array blocks with da.blockwise instead cuts the number of tasks. A 1000 × 1000 array rechunked into 500 × 500 blocks has four blocks to process. section 10.3.2
pos_array = da.blockwise(
lambda x: block_prepare_pos_array(size, x),
'ij', range_array, 'ij', dtype=np.complex128)
pos_array.visualize("10-blockwise.png", rankdir="TB")
This blockwise call applies position preparation to array chunks; visualizing pos_array lets you inspect the resulting task structure rather than assuming array assignment is cheap. section 10.3.2
A scheduler evaluates the graph across computational resources. Dask data frames default to a threaded scheduler; a single-threaded scheduler simplifies debugging, while the distributed scheduler can run on a local machine or across multiple machines. Graph inspection shows dependencies and potential bottlenecks, but the cost of communication depends on the execution architecture. section 10.3 section 10.2.3
