Decode phase
The decode phase is the autoregressive inference stage in which a model generates output tokens sequentially from previously processed context.
Quantization and Fast Inference
Decode is autoregressive inference that generates output tokens one at a time using the preceding context.
- How it is used
- Frames decode as a bandwidth-bound inference workload for comparing CPU and GPU throughput and cost per generated token.
- How deep it goes
- Measures weight traffic, throughput, and hardware effects across formats and builds, without detailing the full autoregressive algorithm.
- Examples
- Compares Llama-2-7B Q4_K_M costs on CPU and L4 instances and decode throughput across five formats and three builds.
- What you can do after
- Evaluate decode deployments using measured throughput, memory bandwidth, and cost per million generated tokens.
Read how Quantization and Fast Inference explains decode phase →
CUDA for LLMs
Decode is the inference phase that generates tokens sequentially from a model's existing context.
- How it is used
- Frames autoregressive inference as sequential token generation with a memory-bandwidth bottleneck.
- How deep it goes
- Traces cached one-token tensor flow and GEMV kernel tradeoffs, without detailing a complete generation loop.
- Examples
- Uses a [B, 1, embed_dim] decode input and compares GEMM and GEMV query-projection timings.
- What you can do after
- Apply KV caching to reuse prior keys and values during sequential token generation.
Also covered in
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
What to know first
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- attention headAn attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.
- memory bandwidthMemory bandwidth is the rate at which data can be transferred between a memory system and a processor or other computing component.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
Related concepts
- prefillcontrasts with · Prefill is the initial inference phase in which a model processes the available input context and prepares state for subsequent autoregressive generation.
- AMX-INT8contrasts with · AMX-INT8 is an Intel Advanced Matrix Extensions execution capability for integer matrix computation using tile registers.
- key-value cacheuses · A key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.
- matrix-vector multiplicationuses · Matrix-vector multiplication is the linear algebra operation of multiplying a matrix by a compatible vector to produce a vector of row-wise dot products.
- memory bandwidthuses · Memory bandwidth is the rate at which data can be transferred between a memory system and a processor or other computing component.
- Apple MLXimplemented by · Apple MLX is an open-source array and machine-learning framework designed for efficient computation on Apple silicon.
Questions about decode phase
What is decode phase?
The decode phase is the autoregressive inference stage in which a model generates output tokens sequentially from previously processed context.
Which Manning books explain decode phase?
Quantization and Fast Inference and CUDA for LLMs.
What should I know before learning decode phase?
Manning books that teach decode phase usually introduce attention, attention head, memory bandwidth and transformer architecture first.
What is the difference between decode phase and prefill?
The decode phase is the autoregressive inference stage in which a model generates output tokens sequentially from previously processed context. Prefill is the initial inference phase in which a model processes the available input context and prepares state for subsequent autoregressive generation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

