liveBook · Manning

Decode phase

The decode phase is the autoregressive inference stage in which a model generates output tokens sequentially from previously processed context.

How 2 Manning books explain it

Quantization and Fast Inference

Decode is autoregressive inference that generates output tokens one at a time using the preceding context.

How it is used
Frames decode as a bandwidth-bound inference workload for comparing CPU and GPU throughput and cost per generated token.
How deep it goes
Measures weight traffic, throughput, and hardware effects across formats and builds, without detailing the full autoregressive algorithm.
Examples
Compares Llama-2-7B Q4_K_M costs on CPU and L4 instances and decode throughput across five formats and three builds.
What you can do after
Evaluate decode deployments using measured throughput, memory bandwidth, and cost per million generated tokens.

Read how Quantization and Fast Inference explains decode phase →

CUDA for LLMs

Decode is the inference phase that generates tokens sequentially from a model's existing context.

How it is used
Frames autoregressive inference as sequential token generation with a memory-bandwidth bottleneck.
How deep it goes
Traces cached one-token tensor flow and GEMV kernel tradeoffs, without detailing a complete generation loop.
Examples
Uses a [B, 1, embed_dim] decode input and compares GEMM and GEMV query-projection timings.
What you can do after
Apply KV caching to reuse prior keys and values during sequential token generation.

Read how CUDA for LLMs explains decode phase →

Also covered in

What to know first

Related concepts

Questions about decode phase

What is decode phase?

The decode phase is the autoregressive inference stage in which a model generates output tokens sequentially from previously processed context.

Which Manning books explain decode phase?

Quantization and Fast Inference and CUDA for LLMs.

What should I know before learning decode phase?

Manning books that teach decode phase usually introduce attention, attention head, memory bandwidth and transformer architecture first.

What is the difference between decode phase and prefill?

The decode phase is the autoregressive inference stage in which a model generates output tokens sequentially from previously processed context. Prefill is the initial inference phase in which a model processes the available input context and prepares state for subsequent autoregressive generation.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.