Throughput measures how much inference work is completed per unit of time, such as generated tokens or served requests per second. In large-language-model deployment, inference optimization raises throughput without changing the model architecture by improving operation scheduling, memory access, and GPU utilization. section 9.3
Autoregressive generation limits throughput because each token depends on earlier tokens. KV caching avoids recomputing the full attention context: previously calculated key and value vectors are stored and reused, while the query for the new token is recomputed. This reduces repeated work, although the cache consumes more memory as sequences and batch sizes grow. section 9.3
GPU memory behavior also determines achievable throughput. Tiling reuses data in fast shared memory, while coalesced access lets warp threads read contiguous addresses and reduces wasted memory transactions. Poor layouts, transpositions, or scattered accesses can instead make memory bandwidth the bottleneck. section 9.4
Measuring and improving inference throughput
A practical benchmark compares generation with and without KV caching for a Llama 3.2 3B Instruct model by recording inference durations. A deployment can then expose a Gemma 3 model through vLLM, while GPU-level improvements target memory layout, tiling, and thread access patterns. section 9.4
