A business-critical pipeline needs to report failures promptly. In Airflow, a callback is a Python function configured to run when a specified DAG or task event occurs. You implement the function; Airflow invokes it when the event happens. section 15.8
Choose the event and the level at which to configure it:
- Set a DAG’s `on_failure_callback` to respond when a DAG run fails. section 15.8
- Set `on_failure_callback` on a task to respond to its failure, or put it in the DAG’s `default_args` to apply it to all tasks. Tasks can also use `on_success_callback` and `on_retry_callback`. section 15.8
Listing 15.15 Defining a failure callback function to execute on task failure
def send_error(): #1
print("ERROR!")
dag = DAG(
dag_id="03_task_failure_callback",
default_args={"on_failure_callback": send_error}, #1
on_failure_callback=send_error, #2
...
)
failing_task = BashOperator(
task_id="failing_task",
bash_command="exit 1", #3
dag=dag,
)
The example assigns `send_error` at both levels: a DAG-run failure can call it, while `default_args` gives every task a failure callback. section 15.8
A failure callback could send a Slack message, report an incident to PagerDuty, or send email. For a straightforward failure email, `email_on_failure` avoids writing a notification function, but Airflow must have SMTP configured. section 15.8
