In this appendix, we’ll provide a brief overview of how to set up a basic monitoring system for our Kafka services using Prometheus (https://prometheus.io/) and Grafana (https://grafana.com/). While this isn’t a dedicated monitoring guide, it will give you a foundational understanding of how to implement monitoring for your Kafka environment.
B.1 Prometheus
Prometheus is an open source system for monitoring and alerting. It periodically scrapes the metric endpoints of the targets to be monitored and stores, in addition to the metric name, a set of labels for identification in its time-series database.
Prometheus also supports service-discovery mechanisms to automatically find scrape targets. Additionally, there’s an option to send metrics directly to Prometheus. Data can then be queried and aggregated using its own query language, PromQL. Alerts can also be defined based on PromQL. For visualization, Prometheus provides its own web interface.
To install Prometheus, we simply need to download the version compatible with our operating system from the official Prometheus website (https://prometheus.io/download/) and extract it. A general installation guide and an introduction to Prometheus are also available at https://mng.bz/YD5z.
Before running the Prometheus binary, however, we make a few changes to the default configuration. For this, we adjust the prometheus.yml file (see configuration documentation at https://mng.bz/GeKD):