Prometheus

As explained in Data Pipelines with Apache Airflow, Second Edition

Prometheus is an open-source monitoring and alerting system that collects time-series metrics from instrumented or exported endpoints.

When Airflow runs in production, metrics can reveal failing tasks, slow DAG parsing, and pressure on the systems it depends on. Prometheus stores monitoring metrics as time series and retrieves them by periodically scraping configured targets. Grafana can then query Prometheus to display those metrics in dashboards. section 15.7.3 section 15.7.4 section 15.7.5

Airflow’s StatsD client pushes metrics, but Prometheus pulls metrics and cannot scrape Airflow directly in Prometheus format. Put the Prometheus StatsD exporter between them: Airflow pushes StatsD metrics to the exporter, which translates and exposes them for Prometheus to scrape. section 15.7.1 section 15.7.3

Figure 15.10 Software and steps required for collecting and visualizing metrics from Airflow. Prometheus collects metrics, and Grafana visualizes metrics in dashboards. The Prometheus StatsD exporter translates StatsD metrics to Prometheus’s metrics format and exposes them for Prometheus to scrape.

Follow the path from Airflow through the StatsD exporter and Prometheus to the Grafana dashboard. section 15.7.1

A minimal setup has three checks:

• Run the StatsD exporter with Docker, exposing port 9125/udp for incoming StatsD metrics and port 9102 for scraping. Match the incoming port to AIRFLOW__SCHEDULER__STATSD_PORT, then inspect http://localhost:9102/metrics. section 15.7.3
• Configure a Prometheus scrape job named airflow with the exporter as its target, then run Prometheus. Check http://localhost:9090/targets: an UP target confirms that scraping is working. section 15.7.3
• Add Prometheus at http://localhost:9090 as a Grafana data source. Plot airflow_dag_processing_total_parse_time to watch how long processing all DAGs takes; a large spike may indicate a scheduler or DAG-file problem. section 15.7.4

Select a task metric

Prometheus identifies a metric by its name and key-value labels. A query such as task_duration{task_id="first_task"} selects that metric for one task ID rather than every task. section 15.7.3

Monitoring need not stop at Airflow metrics. Prometheus exporters can expose machine, metastore, Celery, Redis, endpoint, and Kubernetes metrics where relevant. Expect some dashboard delay: Prometheus scrapes at intervals, and Grafana queries it separately. This setup monitors metrics rather than recording individual events or configuring alerts. section 15.7.5 section 15.7.4

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes

Walks through exporter and scrape configuration, target verification, labels, and dashboard setup; leaves alert configuration outside the walkthrough.

Examples

Uses a Docker-based exporter, an airflow scrape target, an UP target check, and a DAG parse-time dashboard.

What it assumes

Introduces push versus pull collection and the exporter bridge before configuring Prometheus; uses Docker for the demonstrated setup.

What you can do after

Operate/configure it to scrape translated Airflow metrics, verify collection, and supply metrics to Grafana.

How it connects

Links StatsD, the Prometheus StatsD exporter, PromQL, Grafana, and system exporters in an Airflow monitoring pipeline.

liveBookmanning.com

Read Data Pipelines with Apache Airflow, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Learn Docker in a Month of Lunches, Second EditionPrometheus is an open-source monitoring system and time-series database that collects and queries numerical measurements.
• 

  liveBookmanning.com
  ScyllaDB in ActionPrometheus is an open-source monitoring and alerting toolkit that collects and stores time-series metrics, typically by scraping instrumented endpoints.
• 

  liveBookmanning.com
  LLMs in ProductionPrometheus is an open-source monitoring system that collects time-series metrics and supports querying and alerting.
• 

  liveBookmanning.com
  Spring AI in ActionPrometheus is an open-source monitoring system that collects and stores time-series metrics and supports querying and alerting.
• 

  liveBookmanning.com
  Machine Learning Platform EngineeringPrometheus is an open-source monitoring system that collects, stores, and queries time-series metrics.
• 

  liveBookmanning.com
  Cloud Observability in ActionPrometheus is an open-source monitoring system and time-series database for collecting and querying metrics.
• 

  liveBookmanning.com
  Effective Platform EngineeringPrometheus is an open-source monitoring system that collects and stores time-series metrics and supports querying and alerting.

Compare how 8 books explain Prometheus →

Related concepts

• pull modeluses · A pull model is a pattern in which a consumer or recipient initiates requests for data, messages, configuration, or work according to its needs or readiness.
• service discoveryuses · Service discovery is the process of identifying available services and obtaining the names, locations, or connection information needed to access them.
• scrapinguses · Scraping is the automated or systematic retrieval and extraction of information from digital sources.
• Grafanaused by · Grafana is an open-source observability and visualization platform for querying, exploring, and displaying data from multiple sources and configuring alerts.
• metricuses · A metric is a quantitative measure used to characterize or assess a property, behavior, performance, or outcome.

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.