liveBook · Manning

GPU

As explained in LLMs in Production

A GPU is a processor designed for highly parallel numerical computation, commonly used to accelerate machine-learning workloads.

A GPU turns highly parallel numerical work into practical LLM training and inference. Matrix multiplication, linear algebra, and tensor operations make it useful when CPU-only execution cannot meet latency requirements, but adding GPUs also adds platform complexity. section 3.2.4

Treat GPU capacity as a production constraint, not merely a faster processor:

  • Expect scarce capacity: on-premise servers may lack GPUs, while cloud provisioning can fail when every GPU of a selected type in a region is occupied. section 3.2.4
  • Adapt the platform around the resource by rebuilding containers, adding GPU-aware metrics, and extending CPU-and-memory-oriented orchestration systems. section 3.2.4
  • Balance memory against speed: larger GPU memory can support more tokens, but added memory layers can reduce computational performance and increase inference latency. section 3.2.6

The faster model that times out

After upgrading from GPT-3 to GPT-4, a team gained a higher token limit but encountered longer inference times and more timeout errors. A less accurate response delivered quickly can be more useful than a more accurate response that never arrives. section 3.2.6

How LLMs in Production covers it

How deep it goes
Explains parallel processing and numerical benefits, then covers memory, provisioning, containers, and metrics; stops short of hardware internals.
Examples
Uses regional GPU exhaustion, CPU-versus-GPU deployment, and a GPT-3-to-GPT-4 migration with increased timeout errors.
What it assumes
Assumes familiarity with LLM workloads and basic platform orchestration; explains GPU needs from CPU limitations and latency.
What you can do after
Operate/configure GPU-backed LLM systems while anticipating scarcity, workspace failures, container changes, and timeout trade-offs.
How it connects
Connects GPUs with CPUs, Kubernetes, tensor operations, matrix multiplication, attention, token limits, and inference latency.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

Compare how 14 books explain GPU →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.