A factory function accepts configuration and constructs recurring Airflow task structures or complete DAGs. It suits related data sources that share fetch, preprocessing, and export steps while differing in paths or scripts. section 12.1.5
Use factories to control repetition at two levels:
- Expose multiple generated DAGs with unique global names; otherwise later objects overwrite earlier ones and Airflow sees only the last instance. section 12.1.5
Sales and customers ingestion
A single DAG loops over sales and customers, passing each dataset to generate_tasks with separate raw, processed, and output directories and a dataset-specific preprocessing script. A DAG factory can instead create separate DAGs such as 02_dag_factory_sales and 02_dag_factory_customers.
