Suppose a daily pipeline must load data, transform it, and update a model. Apache Airflow coordinates those operations as a workflow: it decides when tasks should run and which tasks must finish before others begin. The work can take place in external systems; Airflow orchestrates it rather than replacing those systems. section 1.0 section 8.0
Define a workflow as a directed acyclic graph (DAG). Tasks are its nodes, and directed edges express dependencies. A Python DAG file typically declares the tasks, their relationships, and execution metadata. Airflow parses that file to discover the workflow. Code-defined DAGs offer flexibility, including generation from configuration, but complex DAGs need careful organization to remain maintainable. section 1.2.1 section 1.3.2
Once a DAG is defined, several components turn it into scheduled work:
- The DAG processor parses DAG files and stores their serialized structure in the metastore. section 1.2.3
- The scheduler checks when a DAG is due and queues tasks according to their dependencies; workers execute the tasks. section 1.2.3
- The triggerer checks completion conditions for supported asynchronous tasks. The API server provides the interface for viewing runs and mediates access to the metastore. section 1.2.3
A DAG can run hourly, daily, or on a cron-like schedule; event-triggered and irregular workflows are also possible. Airflow is particularly useful for recurring, batch-oriented pipelines, integrations with systems such as PostgreSQL and BigQuery, and backfilling earlier executions after changes. It is not designed to process individual streaming events in real time. section 1.2.3 section 1.3.1 section 1.3.2
Start locally, then investigate a failed run
In a Python virtual environment, install `apache-airflow` with `pip install apache-airflow`, then run `airflow standalone`. Open the local UI at `http://localhost:8080` using the generated admin password. The graph view shows task dependencies; the grid view shows current and historical results. If a network interruption makes a download task fail, inspect its logs, fix the connection, and clear the failed task and any downstream tasks that need rerunning. Previously successful upstream tasks need not run again. section 2.3.1 section 1.2.4 section 2.5
For more involved local setups, Docker Compose can run Airflow’s services alongside their dependencies. Production operation adds decisions about scaling, monitoring, access control, and secret encryption. A shared instance uses resources efficiently but can create ownership and failure-isolation problems; separate team instances offer isolation at the cost of more installations to manage. section 2.3.2 section 15.0 section 15.9 section 16.2
