liveBook · Manning

Scheduler

As explained in Fast Python: High performance techniques for large datasets

A scheduler is a system component that selects, orders, dispatches, or assigns execution resources to tasks according to timing, dependency, placement, or other scheduling criteria.

Dask builds a task graph lazily; a scheduler evaluates its nodes across computing resources. If no scheduler is configured, Dask selects a default based on the collection. A Dask data frame, for example, uses the multithreaded scheduler by default. section 10.3

import dask
from dask.base import get_scheduler
import dask.dataframe as dd
 
 
df = dd.read_csv("FY2016-STC-Category-Table.csv")
print(get_scheduler(collections=[df]).__module__)

Use get_scheduler on a collection to inspect which scheduler will execute its task graph. For this data frame, the reported module is dask.threaded. section 10.3

Choose the execution setup to suit the job:

  • Use the single-threaded scheduler for debugging or profiling: sequential execution removes concurrency from the investigation. section 10.3
  • Use the distributed scheduler to run on one machine or across machines. On one machine it supports threads or multiple processes; deployments also include HPC clusters, SSH connections, and cloud providers. section 10.3
  • Treat worker count and threads per worker as workload choices, not universal defaults. NumPy-heavy work may benefit from letting NumPy manage threading rather than adding worker-level threads. section 10.3.1

In dask.distributed, one centralized scheduler assigns tasks to workers, which execute them. A client connects to the scheduler to submit work and inspect the deployment. The scheduler’s dashboard shows computation performance; workers have their own dashboards and monitoring processes. section 10.3.1

dask-scheduler --port 8786 --dashboard-address 8787

Start a scheduler with an explicit task port and dashboard address, then start workers against that port. Running every component locally makes the architecture visible without requiring a cluster. section 10.3.1

A scheduler in concurrency can also mean the mechanism behind preemption: interrupting one task so another can run. Do not assume that Dask’s choice of where to evaluate task-graph nodes describes a preemption policy. section 3.0 section 10.3

How Fast Python: High performance techniques for large datasets covers it

How deep it goes
Explains scheduler alternatives and the distributed scheduler’s components and setup, without detailing its task-placement algorithm.
Examples
Uses a CSV-backed Dask data frame to inspect its default scheduler and command-line examples to launch a local distributed deployment.
What it assumes
Assumes familiarity with Python data libraries and leaves basic multithreading and multiprocessing instruction to other resources.
What you can do after
Operate/configure a local dask.distributed scheduler and workers, inspect their dashboards, and select sequential execution for debugging.
How it connects
Connects scheduling to task graphs, workers, clients, threading, multiprocessing, and preemption.
Cover of Fast Python: High performance techniques for large datasets
liveBookmanning.com
Read Fast Python: High performance techniques for large datasets on liveBook Get the book at manning.com

Also explained in

Compare how 12 books explain scheduler →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fast Python: High performance techniques for large datasets. The section links point to the parts of the book it draws on.