A daily data pipeline should run only after its scheduled interval ends, and each task should wait for its upstream tasks. In Airflow, the scheduler makes those decisions. The DAG processor extracts a DAG’s tasks, dependencies, and schedule; the scheduler uses that information to decide what can run. Workers perform the actual work. section 1.2.3 section 15.1
For each task instance, scheduling has two distinct decisions:
- Ready to schedule? Periodic checks consider conditions such as the end of the schedule interval, satisfied upstream dependencies, and a successful previous run when `depends_on_past=True`. An eligible instance enters the `scheduled` state. section 15.4.1
- Ready to queue? Another check considers available slots and task priority (`priority_weight`). The scheduler then queues a command; the executor takes responsibility for starting the task on a worker. section 15.4.1
Figure 15.8 The ideal flow of a task and the task state for which the components of the scheduler are responsible. The dotted line represents the full scheduler responsibility.
Follow the task-state flow to see where scheduling ends and execution begins. section 15.4.1
What reaches the executor
A queued task can be run with a command shaped like `airflow tasks run [dag_id] [task_id] [execution date]`. The scheduler decides when to queue it; the executor and worker handle its execution. section 15.4.1
For more capacity or redundancy, run another scheduler with `airflow scheduler`. Multiple schedulers coordinate through database row-level locking rather than a separate leader-election tool. The database must support the required locking; disabling row-level locking rules out multiple schedulers. section 15.4.2
When scheduling appears stalled, inspect scheduler logs under `$AIRFLOW_HOME/logs` for repeated checks and scheduling decisions. In Airflow 3, DAG parsing and serialization belong to the separate DAG processor, not the scheduler. section 15.6.2 section 15.4
