A Kafka cluster can survive a broker failure, but prolonged failures erode its remaining fault tolerance. Alerting turns monitoring data into automated notifications so operators can investigate anomalies before they become larger disruptions. An alert fires when a metric, or an aggregation of metrics, crosses a predefined threshold. section 15.0 section 15.4
Build alerts around conditions that need a response, not every metric on a dashboard:
- Visualize a metric before choosing its threshold. If it fluctuates sharply, aggregate it over a longer period: a brief CPU spike to 100% need not wake someone, while sustained high utilization may warrant attention. section 15.4.1
- Start with conservative thresholds and adjust them if false alarms recur. Allow time for self-healing systems, such as Kubernetes restarting a failed container, before escalating; review the failure afterward. section 15.4.1
- Keep diagnostic metrics even when they make poor alert triggers. If consumer lag triggers an alert, produced-record counts can help distinguish a consumer problem from an unusually large influx of records. section 15.4.1
Make each notification useful under pressure. Include the affected metric, what the condition means, and links to relevant dashboards. Where practical, attach a troubleshooting playbook and update it after incidents. Some alerts can also trigger automatic handling, such as a service restart or scaling action. section 15.4.2
Alert conditions depend on where Kafka runs. In Kubernetes, watch both Kafka metrics and pod health, node resource usage, and cluster performance; critical thresholds can prompt a timely response to resource contention. For network failures, quick detection matters, but Kafka-side mitigation may be limited, so clear procedures and network redundancy remain important. section 15.5.4 section 16.1.1
