liveBook · Manning

Speculative decoding

Speculative decoding generates candidate tokens with a faster proposal process and verifies them with a target model before accepting them.

How 3 Manning books explain it

LLMs in Production

Speculative decoding accelerates generation by having a smaller assistant model propose tokens that a larger model verifies in parallel.

How it is used
Accelerates large-model generation when latency matters and additional memory can hold a smaller assistant model.
How deep it goes
Explains assistant proposals and parallel verification, then demonstrates model loading without deriving the acceptance algorithm.
Examples
Uses Whisper-Large-v3 with Distil-Whisper-Large-v3 on an English audio dataset and provides a Transformers implementation.
What you can do after
Apply speculative decoding through Transformers by loading a large model, an assistant model, and generation settings.

Read how LLMs in Production explains speculative decoding →

Rearchitecting LLMs

Speculative decoding is a generation technique in which a smaller model proposes tokens that a larger model verifies, preserving the larger model’s output while potentially accelerating generation.

How it is used
Accelerates a specialist model’s agent-call generation without changing its greedy output.
How deep it goes
Explains draft proposals, parallel target verification, correction, and greedy-output guarantees, while noting rare fp16 ties.
Examples
Uses a five-token ZIP-code illustration and compares target-only generation with a speculative tool-call output.
What you can do after
Apply Hugging Face generation with an assistant model to compare speculative output and timing against target-only generation.

Read how Rearchitecting LLMs explains speculative decoding →

Domain-Specific Small Language Models

An inference-time optimization that predicts multiple future tokens in parallel using draft proposals instead of generating strictly one token at a time.

How it is used
Speeds local LLM generation by using draft proposals that a larger target model verifies.
How deep it goes
Explains draft proposals, target-model verification, shared tokenizers, and the conditional speed benefit.
Examples
Pairs Llama 3.2 1B Instruct as draft with Llama 3.2 3B Instruct as target.
What you can do after
Configure speculative decoding in LM Studio with a draft model and a target model.

Read how Domain-Specific Small Language Models explains speculative decoding →

Also covered in

What to know first

Related concepts

Questions about speculative decoding

What is speculative decoding?

Speculative decoding generates candidate tokens with a faster proposal process and verifies them with a target model before accepting them.

Which Manning books explain speculative decoding?

LLMs in Production, Rearchitecting LLMs and Domain-Specific Small Language Models.

What should I know before learning speculative decoding?

Manning books that teach speculative decoding usually introduce GGUF, language model, transformer block and generation first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.