liveBook · Manning

Memory headroom

As explained in Quantization and Fast Inference

Memory headroom is the unused memory capacity available before a system reaches its memory limit.

Memory headroom is the unused capacity between a workload’s current memory use and the system limit. In LLM serving, memory freed by smaller weights can be assigned to the KV cache, potentially supporting more concurrent requests before they queue. section 12.3.2

When extra headroom makes no difference

On a 24 GB L4, a 3B workload leaves both FP16 and AWQ with ample capacity: KV-cache occupancy stays near 1.3% and the request queue stays empty. The freed-memory advantage is real, but this test does not reach the saturation point where it changes queueing. section 12.3.2

Figure 12.6 KV-cache occupancy (left) and request-queue depth (right) against offered concurrency on a log-2 axis, FP16 versus AWQ. FP16 is the solid line with circle markers, AWQ the dashed line with square markers. Occupancy stays below 2% and queue depth stays at zero across the entire sweep for both formats — the system never approaches its memory limit at this model size, so the saturation knee the freed memory is meant to push outward does not appear.

Compare KV-cache occupancy and request-queue depth for FP16 and AWQ across offered concurrency; neither approaches saturation at this model size. section 12.3.2

How Quantization and Fast Inference covers it

How deep it goes
Measures the effect through KV-cache occupancy and queue depth, but finds no saturation knee at the tested model size.
Examples
Compares FP16 and AWQ at 3B on a 24 GB L4, finding about 1.3% KV-cache occupancy and no queued requests.
What you can do after
Evaluate whether a quantization-related capacity gain is observable under the measured workload.
How it connects
Connects memory headroom to weight quantization, KV cache, vLLM scheduling, and request queues.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.