Suppose a Python DAG generates tasks from a YAML file. Changing the YAML can change the workflow even when the Python file stays untouched. The DAG processor manager periodically reevaluates files in the configured dags directory so Airflow can pick up such changes. section 15.5
The DAG processor reads DAG files, extracts each DAG’s tasks, dependencies, and schedule, then persists a serialized version in the Airflow metastore through the API server. The scheduler uses the parsed DAGs to decide when to queue tasks; workers execute them. Parsing a DAG is therefore distinct from scheduling or running it. section 1.2.3 section 15.1
When DAGs appear slowly or parsing consumes substantial CPU, check the manager’s processing settings:
- AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL sets a minimum time before refreshing the dags-folder file list. Lowering it can help newly added DAGs appear sooner. section 15.5
- AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL sets a lower bound on how often a file is processed; it does not guarantee that frequency. section 15.5
- AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES caps the number of processes used to parse DAG files. More frequent processing improves change detection but costs CPU. section 15.5
Listing 15.5 Example output of DAG processor manager
===========================================================================
[2025-01-24T10:02:53.933+0000] {manager.py:821} INFO - Searching for files
➥in /opt/airflow/dags
[2025-01-24T10:02:53.946+0000] {manager.py:824} INFO - There are 5 files in
➥/opt/airflow/dags
[2025-01-24T10:02:55.984+0000] {manager.py:997} INFO –
➥===============================================================
DAG File Processing Stats
File Path PID Runtime # DAGs # Errors Last Runtime Last Run
--------- --- ------- ------ -------- ------------ --------
.../dag1.py 159 0.01s 1 0 0.06s ... 10:02:25
.../dag2.py 160 0.00s 1 0 0.06s ... 10:02:25
.../dag3.py 1 0 0.04s ... 10:02:25
.../dag4.py 1 0 0.04s ... 10:02:25
.../dag5.py 1 0 0.04s ... 10:02:25
The manager’s file-processing statistics show per-file runtimes, DAG counts, errors, and last-run times—useful evidence when investigating parsing delays. section 15.5
These intervals are boundaries, not promises about exactly when a file will be processed. Choose them with the number and size of DAGs and available resources in mind; expensive DAG-generation code can also slow parsing. section 15.5
