Memory headroom is the unused capacity between a workload’s current memory use and the system limit. In LLM serving, memory freed by smaller weights can be assigned to the KV cache, potentially supporting more concurrent requests before they queue. section 12.3.2
When extra headroom makes no difference
On a 24 GB L4, a 3B workload leaves both FP16 and AWQ with ample capacity: KV-cache occupancy stays near 1.3% and the request queue stays empty. The freed-memory advantage is real, but this test does not reach the saturation point where it changes queueing. section 12.3.2
Figure 12.6 KV-cache occupancy (left) and request-queue depth (right) against offered concurrency on a log-2 axis, FP16 versus AWQ. FP16 is the solid line with circle markers, AWQ the dashed line with square markers. Occupancy stays below 2% and queue depth stays at zero across the entire sweep for both formats — the system never approaches its memory limit at this model size, so the saturation knee the freed memory is meant to push outward does not appear.

Compare KV-cache occupancy and request-queue depth for FP16 and AWQ across offered concurrency; neither approaches saturation at this model size. section 12.3.2
