liveBook · Manning

Backfilling

As explained in Data Pipelines with Apache Airflow, Second Edition

Backfilling is the recomputation or execution of previously scheduled workflow periods after data or processing logic changes.

Suppose a daily workflow analyzes website events, but it starts after several days of events have accumulated. Backfilling runs the DAG for past schedule intervals so those missing days can be processed. Each run handles its own interval rather than reprocessing the entire dataset. section 3.0 section 1.2.5

Set a DAG start date in the past and explicitly enable catchup to create historical runs. Airflow 3 disables catchup by default; Airflow 2 enables it by default. Setting it explicitly avoids a change in behavior when moving between versions. section 3.6 section 3.8

Reprocess a changed calculation

If calc_statistics gains a new statistic, clear its past task runs to recalculate historical results with the new code. This can work even when the original event API retains only seven days of history, provided earlier runs stored the needed data partitions. Loading older events directly from that API remains limited to seven days. section 3.6

Plan for load before enabling a large backfill: Airflow may create hundreds of parallel DAG runs, crowding out other work or overloading external systems. Pools and DAG-level limits such as max_active_tasks and max_active_runs can constrain concurrency. Tasks should also be safe to rerun; idempotency means unchanged inputs do not change the result. section 3.6 section 3.8

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes
Explains start dates, catchup, rerunning cleared tasks, source-data limits, and concurrency risks; defers pool details.
Examples
Uses website-event analysis, a seven-day API history limit, and a changed statistics task to illustrate historical runs.
What it assumes
Assumes basic familiarity with DAGs and scheduled intervals, then develops their use for incremental and historical processing.
What you can do after
Operate/configure it by enabling catchup, clearing past task runs, and limiting concurrent backfill work.
How it connects
Connects backfilling to DAG scheduling, incremental processing, catchup, pools, and idempotency.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.