HorizontalPodAutoscaler

As explained in LLMs in Production

A horizontal pod autoscaler is a Kubernetes controller that adjusts an application's pod replica count according to observed metrics.

Autoscaling can prevent overload, but scaling down also frees resources and cuts costs. A Kubernetes HorizontalPodAutoscaler (HPA) adjusts an application's replica count based on CPU or memory usage. section 6.2.2

Figure 6.6 Basic autoscaling using the in-built k8s horizontal pod autoscaler (HPA). The HPA watches CPU and memory resources and will tell the deployment service to increase or decrease the number of replicas.

The HPA watches CPU and memory and directs the deployment to increase or decrease replicas. section 6.2.2

For LLM workloads, CPU and memory alone may not capture demand; GPU utilization can be a more relevant scaling metric. Using custom metrics requires additional services, such as DCGM for GPU metrics, Prometheus for monitoring, and a custom metrics API such as KEDA. section 6.2.2 section 6.2.2

Figure 6.7 Autoscaling based on a custom metric like GPU utilization requires several extra tools to work, including NVIDIA’s DCGM, a monitoring system like Prometheus Operator, and a custom metrics API like that provided by KEDA.

Custom-metric scaling adds a metrics exporter, monitoring system, and custom metrics API to the basic HPA setup. section 6.2.2

How LLMs in Production covers it

How deep it goes

Explains native CPU and memory scaling, then identifies GPU-based scaling as a custom-metrics setup requiring additional services.

Examples

Uses CPU and memory for basic scaling and GPU utilization as an LLM-serving custom metric.

What you can do after

Evaluate whether native CPU and memory metrics suffice or custom GPU metrics are needed for scaling.

How it connects

Connects Kubernetes autoscaling with CPU and memory metrics, GPU utilization, DCGM, Prometheus, and KEDA.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Kubernetes for DevelopersA pod autoscaler is a Kubernetes controller that adjusts the number of workload replicas according to resource or application metrics.
• 

  liveBookmanning.com
  Server-Side WebAssemblyThe Horizontal Pod Autoscaler is a Kubernetes controller that adjusts a workload's replica count according to observed resource utilization or other metrics.

Compare how 3 books explain HorizontalPodAutoscaler →

Related concepts

• autoscalingimplements · Autoscaling is the automatic adjustment of computing resource capacity or running instances in response to workload demand or measured system conditions.
• Kubernetes Deploymentused by · A Kubernetes Deployment is a declarative API resource that manages the desired state, scaling, and updates of a replicated application workload.
• Kubernetesused by · Kubernetes is an open-source platform for orchestrating and automating the deployment, scaling, and management of containerized workloads across computing clusters.
• replica countconfigures · A replica count is a configuration value specifying the desired number of concurrently running instances of a workload.
• metrics-serveruses · Metrics Server is a Kubernetes component that collects resource-usage metrics from kubelets and exposes them through the resource metrics API.
• replicaconfigures · A replica is a copy of data or state, or an equivalent running instance of a component, maintained alongside others in a replicated system.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.