A Kafka cluster can appear healthy while clients fall behind, or a slow client can appear faulty when its broker has run out of capacity. Monitor the infrastructure, brokers, and clients together so you can locate the constraint before it becomes a service disruption. section 15.1 section 15.3
Build a view from the hardware up:
- Track CPU, memory, disk, and network use. Leave room for partition rebalancing and broker failures. Disk and network alerts around 60% provide planning time, though the right disk threshold depends on how quickly capacity can be added. section 15.1
- Check broker health and load. Under-replicated partitions and partitions below the minimum in-sync replica count should ideally be zero. Request, rejection, and network-processor metrics can reveal bottlenecks or uneven load. section 15.6
- Check whether clients can do their jobs. Producer errors and request latency expose publishing problems; consumer lag and rebalances expose consumption slowdowns. Kafka Connect connector metrics and Kafka Streams processing metrics add application-specific signals. section 15.6 section 15.3
A busy broker is not always a failing broker
Suppose CPU and network use rise during partition rebalancing. A single reading does not establish a fault: temporary resource growth is expected. Sustained high use, by contrast, can indicate growing demand and a need to scale. Pair thresholds with duration and context so alerts point to problems rather than routine activity. section 15.1 section 15.6
Operational responsibility changes with the deployment. On company-owned hardware, include disk health and the physical network. In the cloud, watch latency, throughput, instance and storage performance, and autoscaling. With managed Kafka, the provider handles infrastructure, but you still need to monitor throughput, lag, latency, partition distribution, and service limits. section 15.5.1 section 15.5.3 section 15.5.5
Use alerts for early detection, not every fluctuation: choose meaningful thresholds, smooth noisy metrics, and include context that helps someone respond. Keep monitoring in place when applying client quotas, which can otherwise cause significant problems unnoticed. section 15.6 section 13.3
