liveBook · Manning

CUDA toolkit

As explained in LLMs in Production

The CUDA Toolkit is NVIDIA's software development environment for developing and running GPU-accelerated applications using CUDA.

A GPU can be correctly installed and still deliver unusable performance when software components disagree. Multiple CUDA Toolkit versions once caused PyTorch to select an incompatible version, turning expected model training into an estimated three-year job. section 5.1.2

The CUDA Toolkit provides the software layer that lets deep-learning frameworks communicate with NVIDIA GPUs. In practice, its surrounding stack also supports load balancing, quantization, parallelization, and efficient movement of matrix computations. section 5.1.2

Treat CUDA Toolkit configuration as an operational dependency:

  • Configure the toolkit before installing DeepSpeed or other GPU-oriented libraries, and install a stable, non-nightly PyTorch version first. section 5.1.2
  • Track which CUDA installation PyTorch and build tools actually select; having several versions installed can silently produce incompatibility or severe slowdowns. section 5.1.2
  • Expect Linux configuration to require patience, especially when pairing the toolkit with NVIDIA drivers; virtual environments, Docker, and cloud computing belong in the same operational toolkit. section 5.4.5

Selecting CUDA for llama.cpp

When building llama.cpp on Ubuntu for GPU acceleration, CMake can identify the intended toolkit and compiler explicitly, enable cuBLAS, and target a specific CUDA architecture. This avoids relying on whichever installation Make or the environment discovers by default. section 11.1.3

Table 5.1 Comparison of optimization packages for ML
Library
Faster training or inference
Code integration
Lower accuracy
Many GPUs
Quantization
Optimizations
DeepSpeed
Both
CLI
Depends
Yes
Supports
Caching, gradient checkpointing, memory management, scaling
Accelerate
Both
CLI and Code
Depends
Yes
Supports
Automation, compiling, parallelization
BitsandBytes
Both
Code
Always
NA
Yes but only
Quantization, quantized optimizers
xFormers
Training
Code
Depends
NA
Yes and more
Efficient attention, memory management

Compare the optimization libraries that sit alongside CUDA-enabled training and inference stacks, including their integration style, quantization support, and multi-GPU behavior. section 5.1.2

How LLMs in Production covers it

How deep it goes
Explains compatibility failures and explicit build configuration, but stops short of detailing CUDA internals or driver architecture.
Examples
Uses a PyTorch version conflict and a CMake llama.cpp build specifying nvcc, cuBLAS, toolkit root, and CUDA architecture.
What it assumes
Assumes familiarity with Linux, command-line tools, Python environments, package installation, and compiled builds.
What you can do after
Operate/configure GPU development environments and diagnose framework or build failures caused by mismatched CUDA installations.
How it connects
Connects CUDA Toolkit to PyTorch, NVIDIA drivers, Linux, CMake, cuBLAS, DeepSpeed, Accelerate, BitsandBytes, and xFormers.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.