VRAM is the dedicated memory where a discrete GPU keeps model weights during local inference. Its capacity sets a hard ceiling: an RTX 4060 or 4070 with 8–12GB handles 7B models at Q4, while larger models need a 24GB RTX 4090 or multiple GPUs. section 9.3.2
The main tradeoff is capacity versus throughput. Discrete GPUs provide strong token speed but cannot shift model weights into system RAM when VRAM is exhausted; Apple Silicon shares unified memory instead, handling overflow more gracefully but slowly. section 9.3.2
Use VRAM as an explicit sizing constraint:
- Check model size and quantization before downloading; LLMfit estimates fit, available headroom, and expected throughput from GPU VRAM or unified memory. section 9.3.2
- Treat Q4 as a practical compromise: four-bit quantization reduces parameter storage substantially while sacrificing some accuracy. section 9.3.1
- Reserve discrete-GPU VRAM for the loaded model, while system RAM remains available for the IDE, containers, browser, and test runner. section 9.3.1
A practical capacity check
A 7B model at Q4 fits an 8–12GB GPU and can reach roughly 50–70 tokens per second. A 24GB GPU accommodates larger models, but its VRAM remains a hard limit; compare the model’s requirement with available headroom before committing to the setup. section 9.3.2
Figure 9.3 LLMfit software interface.

Use the LLMfit interface to estimate whether a model and quantization level fit available VRAM and to assess headroom and throughput. section 9.3.2





