A data pipeline can fail even when its task code has not changed: an external service may be unavailable, a network connection may drop, or a disk may break. In Airflow, an unsuccessful task is marked failed. By default, downstream tasks require their preceding tasks to succeed, so a task blocked by a failure is marked upstream_failed rather than run. section 2.5
When a task fails, work from the affected run rather than restarting the whole workflow:
- Find the failed task in the DAG’s graph or grid view. The grid also shows results across current and historical runs. section 1.2.4 section 2.5
- Open its logs and inspect the stack trace for a likely cause. Fix the underlying problem before rerunning it. section 2.5
- Clear the failed task instance to let Airflow run it again. When needed, select downstream tasks in the clear menu; previously successful tasks need not be restarted. section 2.5
Interrupted image download
A network interruption prevents an image-download task from connecting to its URL. The task fails, and its dependent notification task is marked upstream_failed. A ConnectionError in the download task’s logs points to the connection problem. After restoring connectivity, clear the failed task and the downstream work that should run afterward. section 2.5
Configure retries, optionally with waits between attempts, for intermittent task failures. If retries do not resolve the problem, Airflow records the failure and can send a notification when configured. Production monitoring should also track errors beyond individual DAG runs: examples include zombie tasks, timed-out tasks, and web-server responses other than HTTP 200. Errors are one of four monitoring signals alongside latency, load, and saturation. section 1.2.4 section 15.7.5
ConnectionError: Could not connect to https://thespacedevs- ➥prod.nyc3.digitaloceanspaces.com/media/images/ ➥long_march_2_image_20210908195835.jpeg.
The ConnectionError illustrates the kind of concrete clue to look for in a failed task’s logs. section 2.5
