Alertmanager is the notification-routing component in the Prometheus ecosystem. Prometheus evaluates alert conditions, while Alertmanager groups and forwards resulting alerts to channels such as email, Slack, or PagerDuty. section 3.5 section 11.1.4
A production alerting path typically proceeds as follows:
- Expose application metrics through a scrape endpoint such as /metrics, allowing Prometheus to collect service health data. section 3.5
- Define Prometheus alert rules with PromQL expressions, such as detecting an unavailable target with up == 0 or a missing metric with absent(...). section 11.1.4
- Set a pending period, severity label, summary, and description so transient failures do not immediately become notifications and recipients receive useful context. section 11.1.4
- Configure routing with grouping, initial and subsequent notification intervals, repeat intervals, and receivers. section 11.1.4
Route critical service failures to email
A BentoML deployment can raise a critical ServiceDown alert after five minutes when its up metric equals zero. Alertmanager can group alerts by alertname and job, wait 30 seconds before the first notification, repeat notifications every three hours, and send resolved notifications through an SMTP-configured Gmail receiver.
