An LLM service rarely receives a steady stream of requests. Daytime traffic may require more capacity than nighttime traffic, while a sudden burst can threaten the service. Autoscaling adjusts the number of service replicas as demand changes: scale up to protect performance and scale down to release resources and reduce cost. section 6.2.2
Choose a scaling signal that reflects the service’s needs:
- For basic Kubernetes autoscaling, a horizontal pod autoscaler (HPA) increases or decreases replicas using CPU or memory usage. section 6.2.2
- For LLM inference, CPU and memory alone may be insufficient. GPU utilization is a more relevant custom metric, but it requires additional tooling. section 6.2.2
Figure 6.7 Autoscaling based on a custom metric like GPU utilization requires several extra tools to work, including NVIDIA’s DCGM, a monitoring system like Prometheus Operator, and a custom metrics API like that provided by KEDA.
The custom-metric path connects GPU utilization to autoscaling through NVIDIA DCGM, a monitoring system such as Prometheus Operator, and a custom metrics API such as KEDA’s. section 6.2.2
Scaling up helps prevent overload; scaling down is just as important to the business because idle replicas cost money. Neither happens usefully by default: an engineer must identify the service’s capacity needs and configure the scaling signals accordingly. section 6.2.2
