Speculative decoding
Speculative decoding generates candidate tokens with a faster proposal process and verifies them with a target model before accepting them.
LLMs in Production
Speculative decoding accelerates generation by having a smaller assistant model propose tokens that a larger model verifies in parallel.
- How it is used
- Accelerates large-model generation when latency matters and additional memory can hold a smaller assistant model.
- How deep it goes
- Explains assistant proposals and parallel verification, then demonstrates model loading without deriving the acceptance algorithm.
- Examples
- Uses Whisper-Large-v3 with Distil-Whisper-Large-v3 on an English audio dataset and provides a Transformers implementation.
- What you can do after
- Apply speculative decoding through Transformers by loading a large model, an assistant model, and generation settings.
Rearchitecting LLMs
Speculative decoding is a generation technique in which a smaller model proposes tokens that a larger model verifies, preserving the larger model’s output while potentially accelerating generation.
- How it is used
- Accelerates a specialist model’s agent-call generation without changing its greedy output.
- How deep it goes
- Explains draft proposals, parallel target verification, correction, and greedy-output guarantees, while noting rare fp16 ties.
- Examples
- Uses a five-token ZIP-code illustration and compares target-only generation with a speculative tool-call output.
- What you can do after
- Apply Hugging Face generation with an assistant model to compare speculative output and timing against target-only generation.
Read how Rearchitecting LLMs explains speculative decoding →
Domain-Specific Small Language Models
An inference-time optimization that predicts multiple future tokens in parallel using draft proposals instead of generating strictly one token at a time.
- How it is used
- Speeds local LLM generation by using draft proposals that a larger target model verifies.
- How deep it goes
- Explains draft proposals, target-model verification, shared tokenizers, and the conditional speed benefit.
- Examples
- Pairs Llama 3.2 1B Instruct as draft with Llama 3.2 3B Instruct as target.
- What you can do after
- Configure speculative decoding in LM Studio with a draft model and a target model.
Read how Domain-Specific Small Language Models explains speculative decoding →
Also covered in
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
- CUDA for LLMs
What to know first
- GGUFGGUF is a binary file format for storing machine-learning model tensors and associated metadata, designed to support efficient model loading and inference.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- transformer blockA transformer block is a modular unit of a transformer neural network that applies attention and other transformations, commonly including feed-forward computation, to representations.
- generationGeneration is the production of an output by a computational model from supplied input or context, often using patterns learned from data.
Related concepts
- draft modeluses · A draft model is a language model that proposes candidate tokens for a target model to verify during speculative decoding.
- target modeluses · A target model is the model that verifies candidate tokens proposed by a draft model during speculative decoding.
- acceptance ratemeasures · Acceptance rate is the proportion of proposed or offered items that are accepted.
- greedy decodinguses · Greedy decoding generates a sequence by selecting the highest-probability next token at each step.
- tokenizeruses · A tokenizer is a text-processing component or procedure that divides text into discrete tokens and may map them to identifiers according to specified rules or a vocabulary.
Questions about speculative decoding
What is speculative decoding?
Speculative decoding generates candidate tokens with a faster proposal process and verifies them with a target model before accepting them.
Which Manning books explain speculative decoding?
LLMs in Production, Rearchitecting LLMs and Domain-Specific Small Language Models.
What should I know before learning speculative decoding?
Manning books that teach speculative decoding usually introduce GGUF, language model, transformer block and generation first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


