Suppose a daily data pipeline is enabled after its start date. Should it run for the dates already missed? The DAG’s `catchup` setting decides whether Airflow schedules those historical runs. With `catchup=False`, it considers future time points; with `catchup=True`, it also schedules past time points starting at `start_date`. section 3.2
Set `catchup` explicitly. Airflow 3 defaults to `False`, while Airflow 2 defaults to `True`. In the daily example below, enabling it causes runs for past scheduled dates that would otherwise not execute. section 3.3.1 section 3.6
Listing 3.3 Enabling catchup for past runs
with DAG(
dag_id="02_daily_schedule",
start_date=pendulum.datetime(year=2025, month=1, day=1),
schedule=CronTriggerTimetable("@daily", timezone="UTC"),
catchup=True, #1
):
The DAG configuration sets a January 1, 2025 start date, a daily UTC timetable, and `catchup=True`. section 3.3.1
Plan for the work those extra runs create:
- Check source history. Historical runs cannot load data older than the source retains; the example events API exposes only seven days. section 3.6
- Keep outputs separate by run. Writing every run to one file overwrites earlier results; a filename based on `logical_date` preserves date-specific output. section 3.3.1
- Control load. Catchup can create many parallel runs, crowding out other DAGs or overloading external systems. DAG-level `max_active_tasks` and `max_active_runs` can constrain concurrency. section 3.6
Catchup schedules missing historical runs; it is not itself a way to rerun completed work. To recalculate historical statistics after a code change, clear the relevant past task runs so they can execute again against previously loaded data. section 3.6
