concept dag in category apache airflow

This is an excerpt from Manning's book Data Pipelines with Apache Airflow MEAP V05. Login to get full access to this book.

Figure 1.7 Airflow pipelines are defined as DAGs (Directed Acyclic Graphs) using Python code in DAG files. Each DAG file typically defines one DAG, which describes the different tasks and their dependencies. Besides this, the DAG also defines a schedule interval which determines when the DAG is executed by Airflow.

One advantage of defining Airflow DAGs in Python code is that this programmatic approach provides you with a lot of flexibility for building DAGs. For example, as we will see later in this book, you can use Python code to dynamically generate optional tasks depending on certain conditions, or even generate entire DAGs based on external metadata or configuration files. This flexibility allows a great deal of customization in how you build your pipelines - allowing you to fit Airflow to your needs for building arbitrarily complex pipelines.  

Figure 12.2 By default, Airflow will load example DAGs (and connections, not displayed here)

These examples might come in handy during development, but are likely not desirable for a production system. You can exclude example DAGs by setting AIRFLOW__CORE__LOAD_EXAMPLES=False.

However, upon restarting the scheduler and webserver; you will probably be surprised to still see the DAGs and connections. The reason is that setting load_examples to False tells Airflow to not load example DAGs (does not apply to connections!), and Airflow will not reload example DAGs. However, already loaded DAGs remain in the database and are not deleted.

Listing 12.19

dag = DAG(
   dag_id="task_failure_email",
   default_args={"email": "bob@work.com"},
   ...
)

copy