concept Grafana in category apache airflow
This is an excerpt from Manning's book Data Pipelines with Apache Airflow MEAP V05.
Why the mixing and matching? Mainly because Prometheus is the tool of choice for many developers, sys admins, and more for monitoring. It is used at many companies, and prevails over StatsD on many points such as its flexible data model, ease of operation, and integration with virtually any other system. Therefore, we also prefer Prometheus for dealing with metrics, and we demonstrate how to transform StatsD metrics into Prometheus metrics, after which we can visualize the collected metrics with Grafana. Grafana is a dashboarding tool for visualizing time series data for monitoring purposes.
The steps from Airflow to Grafana will look as follows:
Figure 12.16 Software and steps required for collecting and visualizing metrics from Airflow. Prometheus collects metrics and Grafana visualizes metrics in dashboards. The Prometheus StatsD exporter translates StatsD metrics to Prometheus’ metrics format and exposes them for Prometheus to scrape.
Let’s set up this system from left (Airflow) to right (Grafana), with the goal of creating a dashboard visualizing metrics from Airflow.
After collecting metrics with Prometheus, the last piece of the puzzle is to visualize these metrics in a dashboard. This should provide us a quick understanding of the functioning of the system. Grafana is the main tool for visualizing metrics. The easiest way to get Grafana up and running is once again with Docker:
On http://localhost:3000, this is the first view of Grafana you will see: