liveBook · Manning

Custom metrics API

As explained in LLMs in Production

The custom metrics API is a Kubernetes API for exposing application-defined metrics associated with Kubernetes resources.

The custom metrics API connects application-defined metrics to Kubernetes autoscaling. Prometheus can collect and store metrics, but an adapter service must expose them to the Horizontal Pod Autoscaler (HPA). section 6.2.2

Production use involves several practical constraints:

  • Kubernetes provides custom.metrics.k9s.io alongside metrics.k8s.io and external.metrics.k8s.io for autoscaling beyond CPU and memory. section 6.2.2
  • Only one adapter API service can be registered at a time for each custom or external metrics endpoint, complicating provider changes. section 6.2.2
  • Prometheus Adapter works with Prometheus but requires metric configuration in a ConfigMap; KEDA offers broader Prometheus-based scaling and scaling to zero. section 6.2.2
Figure 6.7 Autoscaling based on a custom metric like GPU utilization requires several extra tools to work, including NVIDIA’s DCGM, a monitoring system like Prometheus Operator, and a custom metrics API like that provided by KEDA.
figure

Use the architecture diagram to trace GPU-utilization autoscaling through DCGM, Prometheus Operator, and a custom metrics API. section 6.2.2

How LLMs in Production covers it

How deep it goes
Explains the Prometheus-to-adapter-to-HPA path and provider limitation, but does not detail API schemas or request mechanics.
Examples
Uses GPU utilization as the concrete custom metric, with DCGM, Prometheus Operator, and KEDA forming the supporting stack.
What it assumes
Assumes Kubernetes autoscaling and Prometheus concepts, while distinguishing resource, custom, and external metrics.
What you can do after
Operate/configure Kubernetes autoscaling around application-defined metrics through an adapter such as Prometheus Adapter or KEDA.
How it connects
Links Prometheus, Horizontal Pod Autoscaler, external metrics API, Prometheus Adapter, KEDA, and GPU utilization.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.