An Airflow secrets backend lets DAGs retrieve variables and connections from an external secrets-management system without copying secret values into Airflow. This keeps credentials in a system designed for centralized storage, rotation, and versioning while preserving Airflow’s existing variable and connection interfaces. section 16.5
Configure and use a backend around four operational rules:
- Set AIRFLOW__SECRETS__BACKEND and AIRFLOW__SECRETS__BACKEND_KWARGS to identify and configure the integration. section 16.5
- Give secrets the prefix and hierarchical path expected by the selected backend; the Vault example uses paths such as secret/connections/secure_api. section 16.5
- Store connection values under conn_uri and variable values under value. section 16.5
- Leave DAG code focused on Airflow objects: operators continue referring to connection IDs instead of naming the external secrets system. section 16.5
Call an API through Vault
A configured Vault backend can supply the connection behind an HTTP task. The DAG specifies http_conn_id="secure_api" and performs the request without an explicit Vault reference. Airflow resolves the connection through the backend before the operator runs. section 16.5
Listing 16.6 Fetching connection details from a configured secrets backend
import pendulum
from airflow.providers.http.operators.http import HttpOperator
from airflow.sdk import DAG
with DAG(
dag_id="secretsbackend_with_vault",
start_date=pendulum.today("UTC").add(days=-3),
schedule=None,
):
call_api = HttpOperator(
task_id="call_api",
http_conn_id="secure_api", #1
method="GET",
endpoint="",
log_response=True,
)
Inspect the compact DAG pattern that uses a configured secrets backend while keeping the external storage system out of task code. section 16.5
Use a secrets backend when credentials belong in a dedicated external system such as HashiCorp Vault, Azure Key Vault, AWS SSM, or Google Cloud Secrets Manager. Airflow can also centralize credentials in its metastore, but external systems may provide rotation and versioning that Airflow does not. section 12.1.2 section 16.5
