A data pipeline may need configurable input paths and output tables without embedding those values directly in its DAG definition. Airflow variables store such configuration in the Airflow metastore. section 12.1.3
Retrieve a variable by name with Variable.get and pass the result to a task. Variables can be managed through the Airflow UI, environment variables, or the CLI. section 12.1.3
Listing 12.17 Storing configuration options in Airflow Variables
from airflow.sdk import Variable
input_path = Variable.get("dag1_input_path") #1
output_path = Variable.get("dag1_output_path")
fetch_data = PythonOperator(
task_id="fetch_data",
op_kwargs={
"input_path": input_path,
"output_path": output_path,
},
...
)
The listing demonstrates loading input and output paths from Airflow variables and passing them into a PythonOperator. section 12.1.3
Use variables with these operational considerations:
- Avoid fetching variables in the global scope of the DAG; the DAG processor can refetch them from the database whenever it reads the DAG definition. section 12.1.3
- Choose a consistent configuration convention across DAGs; shared configuration belongs in a single location when applying DRY principles. section 12.1.3
- Treat variables as one possible location for secrets, but prefer a dedicated secrets-management system when rotation and versioning are required. section 16.5
