liveBook · Manning

GPU memory

As explained in Vibe Engineering

VRAM is memory on a graphics-processing unit used to store graphics and compute workloads such as model parameters.

VRAM is the dedicated memory where a discrete GPU keeps model weights during local inference. Its capacity sets a hard ceiling: an RTX 4060 or 4070 with 8–12GB handles 7B models at Q4, while larger models need a 24GB RTX 4090 or multiple GPUs. section 9.3.2

The main tradeoff is capacity versus throughput. Discrete GPUs provide strong token speed but cannot shift model weights into system RAM when VRAM is exhausted; Apple Silicon shares unified memory instead, handling overflow more gracefully but slowly. section 9.3.2

Use VRAM as an explicit sizing constraint:

  • Check model size and quantization before downloading; LLMfit estimates fit, available headroom, and expected throughput from GPU VRAM or unified memory. section 9.3.2
  • Treat Q4 as a practical compromise: four-bit quantization reduces parameter storage substantially while sacrificing some accuracy. section 9.3.1
  • Reserve discrete-GPU VRAM for the loaded model, while system RAM remains available for the IDE, containers, browser, and test runner. section 9.3.1

A practical capacity check

A 7B model at Q4 fits an 8–12GB GPU and can reach roughly 50–70 tokens per second. A 24GB GPU accommodates larger models, but its VRAM remains a hard limit; compare the model’s requirement with available headroom before committing to the setup. section 9.3.2

Figure 9.3 LLMfit software interface.

Use the LLMfit interface to estimate whether a model and quantization level fit available VRAM and to assess headroom and throughput. section 9.3.2

How Vibe Engineering covers it

How deep it goes
Explains capacity limits, memory placement, quantization, and sizing tools, but stops short of hardware-level GPU internals.
Examples
Uses RTX 4060, RTX 4070, RTX 4090, 7B Q4 models, and LLMfit as concrete sizing examples.
What it assumes
Assumes familiarity with local models and GPUs while defining VRAM, unified memory, and Q4 quantization from the basics.
What you can do after
Operate/configure local inference by matching model quantization and size to VRAM, headroom, and expected throughput.
How it connects
Connects VRAM with local inference, unified memory, model quantization, system RAM, and GPU throughput.

Also explained in

Compare how 6 books explain GPU memory →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.