An Airflow task that calls a protected API needs a connection, but copying its credentials into Airflow creates another place to manage them. A central secrets manager stores passwords, certificates, and keys for applications to request when needed. Examples include HashiCorp Vault, Azure Key Vault, AWS SSM, and Google Cloud Secrets Manager. section 16.5
Airflow’s secrets backend lets tasks keep using the usual connection and variable classes while retrieving their values from an external system. This consolidates secrets rather than scattering copies across applications. Central systems can also provide rotation and versioning, which Airflow does not provide. section 16.5
To use the Vault example:
- Configure AIRFLOW__SECRETS__BACKEND and AIRFLOW__SECRETS__BACKEND_KWARGS for the backend. The illustrated backend arguments specify a Vault URL, token, and connections path. section 16.5
- Store the connection under a path such as secret/connections/secure_api, with its connection details in the conn_uri key. Store variables under the value key; secrets must share a common prefix. section 16.5
- Keep the DAG’s connection ID as secure_api. The HttpOperator uses that ID to obtain the URL without referring to Vault explicitly in DAG code. section 16.5
Listing 16.6 Fetching connection details from a configured secrets backend
import pendulum
from airflow.providers.http.operators.http import HttpOperator
from airflow.sdk import DAG
with DAG(
dag_id="secretsbackend_with_vault",
start_date=pendulum.today("UTC").add(days=-3),
schedule=None,
):
call_api = HttpOperator(
task_id="call_api",
http_conn_id="secure_api", #1
method="GET",
endpoint="",
log_response=True,
)
The DAG’s HttpOperator uses http_conn_id="secure_api"; the external storage choice stays out of the task definition. section 16.5
For production cloud tasks, prefer a secrets manager to spreading sensitive credentials across systems. It is one layer of Airflow security, alongside access controls, traffic encryption, and encryption of credentials stored in Airflow’s database. section 8.2.3 section 16.6
