Suppose a daily pipeline has been defined as a DAG. Airflow needs a shared record of its tasks, dependencies, schedule, and execution state so it can decide what to run and display progress. The Airflow metastore is the metadata database that holds this information. The DAG processor parses DAG files and stores serialized DAGs there; the scheduler uses the parsed definitions to schedule tasks. section 1.2.3 section 15.1
Keep the metastore’s role distinct from the processes around it:
- The DAG processor reads DAG files and makes their serialized definitions available through the metastore. The metastore does not replace the folder containing those files. section 15.1
- The scheduler determines which tasks to queue; workers perform the work. The metastore holds Airflow metadata, including DAG-run information, rather than executing tasks. section 15.1
- In Airflow 3, components use the API server as the gateway for metastore reads and writes. Avoid direct metastore access from a DAG: it can cause problems for the whole Airflow instance. section 1.2.3 section 15.1
From local setup to a production database
For a local start, `airflow standalone` initializes the metastore, creates a user, and launches the main Airflow processes. For production, an external database lets an organization manage access, backups, and recovery outside the Airflow deployment. In the Kubernetes example, a secret supplies a PostgreSQL connection URL; Helm configuration disables the chart’s PostgreSQL database and points Airflow at that secret. Switching metastores also switches the stored state: the example’s previous admin user is absent from the new database. section 2.3.1 section 17.2.6
Listing 17.12 Using an external database
# Override default values in the Apache Airflow Helm Chart
#
# 3 - Using the external database
postgresql:
enabled: false
data:
metadataSecretName: mydatabase
# 2 - Overriding the webserver secret reference
apiSecretKeySecretName: my-apiserver-secret
# 1 - Overriding the default user since we use a different database
webserver:
# Create initial user.
defaultUser:
enabled: true
role: Admin
username: airflow
email: airflow@example.com
firstName: Airflow
lastName: Second Edition
password: airflow
The Helm values show the two essential deployment changes: disable the bundled PostgreSQL database and set `data.metadataSecretName` to the secret for the external metastore. section 17.2.6
Plan database capacity and operations as part of the Airflow deployment. Distributed Airflow can open many database connections; for PostgreSQL, a connection pooler such as PgBouncer can help manage them. section 17.2.6
