liveBook · Manning

Decode phase

As explained in Quantization and Fast Inference

Decode is autoregressive inference that generates output tokens one at a time using the preceding context.

Decode generates output autoregressively, one token at a time from preceding context. For a 7B model, each generated token requires reading roughly the model’s full weight footprint, making decode bandwidth-bound on the measured CPU host. section 10.4.3

Compare decode deployments by throughput and cost per million generated tokens. In the Llama-2-7B Q4_K_M comparison, single-stream throughput is about 20 tokens/s on a CPU instance and 50 tokens/s on an L4 instance; at saturation, the L4 costs roughly 3.9 times less per token. section 10.1

CPU decode throughput

On a 22-vCPU host, Llama-2-7B F16 decode measures about 6.3 tokens/s, corresponding to roughly 80 GB/s of effective DRAM bandwidth. AVX-512+VNNI improves integer-format decode over AVX-2 by only 1.8–3.7%; the AMX-enabled build is slower across the measured quantized formats. section 10.4.3

Figure 10.1 Effective cost per million Llama-2-7B Q4_K_M tokens versus sustained demand, on Google Cloud c3-standard-22 (CPU-only, solid) and g2-standard-4 with one NVIDIA L4 (dashed) in us-central1 at on-demand list pricing. Each curve declines hyperbolically until the instance's single-stream throughput ceiling (CPU at 20 tok/s, GPU at 50 tok/s; tick marks at successive replica boundaries), then steps up as additional replicas absorb demand above ceiling. Below saturation the ratio between curves equals the ratio of hourly prices (1.57×). At the GPU's saturation the ratio is roughly 3.9×, widening further at higher demand because the CPU instance saturates at 2.5× lower throughput. The L4 wins on cost per token at every demand point in the plot; the gap is bounded between 1.57× and roughly 5×.

Compare estimated cost per million generated tokens across sustained demand, instance throughput ceilings, and replica boundaries. section 10.1

Figure 10.9 Decode throughput on Llama-2-7B across five quantization formats and three llama.cpp builds. F16 is flat across all three builds because it does not enter any of the integer dot-product instructions the build flags govern; its column measures the host's memory bandwidth alone. The integer formats show single-digit-percent gains from AVX-512+VNNI over AVX-2 and consistent regressions on the AMX-enabled build. Q5_K_M's regression on AMX is roughly three times the regressions Q4_K_M and Q8_0 seen on the same build.

Compare measured decode throughput across quantization formats and llama.cpp builds. section 10.4.3

How Quantization and Fast Inference covers it

How deep it goes
Measures weight traffic, throughput, and hardware effects across formats and builds, without detailing the full autoregressive algorithm.
Examples
Compares Llama-2-7B Q4_K_M costs on CPU and L4 instances and decode throughput across five formats and three builds.
What you can do after
Evaluate decode deployments using measured throughput, memory bandwidth, and cost per million generated tokens.
How it connects
Connects autoregressive inference with quantization, memory bandwidth, CPU and GPU hardware, and SIMD instruction sets.

Also explained in

Compare how 2 books explain decode phase →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.