dask.distributed runs Dask tasks through a centralized scheduler and workers. A client submits work and can inspect execution. The same arrangement can run on one machine or across a cluster, but moving data between nodes has a performance cost. section 10.3.1 section 10.4
For an explicit single-machine setup:
- Start a scheduler with `dask-scheduler --port 8786 --dashboard-address 8787`. section 10.3.1
- Start workers with `dask-worker --nprocs auto 127.0.0.1:8786`. Worker processes and threads can also be configured directly; the best mix depends on the workload. section 10.3.1
- Connect with `Client('127.0.0.1:8786')`. Once connected, Dask data-structure calls use the distributed scheduler unless overridden. section 10.3.2
Task granularity matters as much as worker count. Assigning values cell by cell to a Dask array creates a task for each cell: a 1,000 × 1,000 image would create one million tasks. Instead, partition the array and use `da.blockwise` to initialize whole blocks. Use `rechunk`, `persist`, and task-graph visualization to inspect the resulting computation before timing it. section 10.3.2
Mandelbrot image
Build an index array, reshape and rechunk it, then turn each block into complex-plane positions with `da.blockwise`. Apply the point computation to produce image values. A timing function varies chunk count and whether intermediate arrays are persisted, making those execution choices measurable rather than assumed improvements. section 10.3.2
Use the scheduler dashboard to check how computations perform and inspect workers. Each worker also has a dashboard and a nanny that monitors its state. Worker and thread counts need workload-specific testing: NumPy-heavy work may be better left to NumPy's own threading rather than multiplied across many Dask worker threads. section 10.3.1
