liveBook · Manning

Gpu shortage

As explained in LLMs in Production

A GPU shortage is a condition in which demand for available graphics processors exceeds their supply or accessibility.

A GPU shortage occurs when demand for available graphics processors exceeds supply or practical access. Unlike CPU and memory capacity, GPU capacity cannot be assumed to be plentiful: on-premises machines may have none, while cloud regions can exhaust a selected GPU type. Limited regional availability can force deployments farther from users, increasing latency and potentially cost. section 3.2.4 section 6.3.4

Practical responses include:

  • Search across cloud providers, clusters, regions, and zones for capacity that matches the job. section 6.3.4
  • Use GPU pooling, dynamic sharing, job scheduling, oversubscription, or fractional GPUs to improve cluster utilization. section 6.3.4
  • Expect additional platform work: GPU-aware containers, deployment configuration, and metrics may require rebuilding assumptions designed for CPU-only services. section 3.2.4

Provisioning a GPU-backed notebook

SkyPilot can search configured infrastructure and start a two-L4-GPU Jupyter environment with port forwarding: sky gpunode -p 8888 -c jupyter-vm --gpus l4:2 --cloud gcp --region us-west1. section 6.3.4

$ sky gpunode -p 8888 -c jupyter-vm --gpus l4:2 --cloud gcp --region us-west1

Inspect the complete SkyPilot command for provisioning a two-GPU Jupyter node on Google Cloud. section 6.3.4

How LLMs in Production covers it

How deep it goes
Explains scarcity, regional exhaustion, latency, and mitigation tooling without detailing hardware economics.
Examples
Uses on-premises access restrictions, regional exhaustion errors, SkyPilot, Run:ai, and a two-L4-GPU Jupyter command.
What it assumes
Assumes familiarity with cloud or on-premises infrastructure, GPU-backed workloads, and basic deployment resources.
What you can do after
Operate/configure GPU jobs through SkyPilot and evaluate sharing or scheduling tools when capacity is constrained.
How it connects
Connects GPU availability with cloud computing, Kubernetes, GPU scheduling, resource management, and cluster sharing.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.