Anomaly detection identifies observations or behavior that significantly departs from an established normal baseline. For metrics, the baseline describes how a healthy service or system usually behaves, including its expected range, minimum, and maximum. section 2.4.3 section 7.1.1
A baseline can be established by observing normal behavior manually or by automating the process, potentially with machine-learning techniques. When a signal leaves its normal range, detection can feed alerting so someone responsible receives a notification. section 2.6 section 7.1.1
Operational use depends on the signal and the question being asked:
- Define a condition such as a metric exceeding a threshold for at least three minutes, then notify an operator. section 2.4.3
- Use historical metrics and histograms to study capacity, scalability, and changing operational behavior. section 2.4.4
- Choose retention and collection deliberately; indiscriminately keeping every signal increases cost and can reduce return on investment. section 2.6
A service leaving its normal range
Suppose a healthy service normally stays within a known metric range. Observing that range establishes the baseline; an automated detector can flag a departure, and an alert can notify the responsible operator. Prometheus anomaly detector is one implementation option, while CloudWatch can provide a monitoring view for website availability. section 2.6 section 7.1.1
