KEDA

As explained in LLMs in Production

KEDA is an open-source Kubernetes project that scales workloads according to event sources and custom or external metrics.

A Kubernetes service can use its horizontal pod autoscaler (HPA) to adjust replicas based on CPU or memory use. An LLM inference service may also need to scale on GPU utilization, traffic, or queue activity. KEDA (Kubernetes Event-Driven Autoscaling) extends the options for scaling Kubernetes workloads with custom metrics and event-driven signals. section 6.2.2

For GPU-aware scaling, DCGM supplies GPU metrics to Prometheus Operator. KEDA exposes the metrics through a Kubernetes custom metrics API; its ScaledObject creates the HPA used to adjust replicas. Unlike Prometheus Adapter, which requires metric configuration in a ConfigMap, KEDA can use metrics already exposed through the Prometheus API. section 6.2.2

Choose the scaling signal with the zero-replica limit in mind:

• Use traffic or queue signals when turning an idle service off completely matters: KEDA supports scaling to and from zero. section 6.2.2
• Do not rely on CPU, memory, or GPU resource metrics to bring a service back from zero; these cannot trigger scale-from-zero. section 6.2.2

Figure 6.7 Autoscaling based on a custom metric like GPU utilization requires several extra tools to work, including NVIDIA’s DCGM, a monitoring system like Prometheus Operator, and a custom metrics API like that provided by KEDA.

The GPU-utilization autoscaling diagram shows why KEDA needs a metric source and monitoring system rather than replacing them. section 6.2.2

How LLMs in Production covers it

How deep it goes

Explains the metric pipeline, ScaledObject, adapter trade-off, and scale-to-zero limit without providing installation steps or a manifest.

Examples

Illustrates GPU-utilization autoscaling through a DCGM-to-Prometheus-to-KEDA architecture diagram.

What it assumes

Assumes familiarity with Kubernetes workloads while establishing CPU- and memory-based HPA scaling as the baseline.

What you can do after

Evaluate/understand it as an option for GPU-aware autoscaling or traffic- and queue-driven scale-to-zero.

How it connects

Links KEDA to Kubernetes HPA, custom metrics APIs, Prometheus, DCGM, GPU utilization, and queue-based autoscaling.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

• HorizontalPodAutoscaleralternative to · A HorizontalPodAutoscaler is a Kubernetes API resource and controller that adjusts a workload’s replica count according to observed metrics and configured targets.
• scalabilityimplements · Scalability is the ability of a system, process, or method to accommodate increasing workload, size, or demand while maintaining acceptable operation.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.