Speculative decoding
Speculative decoding is an inference technique in which a faster draft process proposes multiple tokens that a larger target model verifies or rejects.
LLMs in Production
Speculative decoding accelerates generation by having a smaller assistant model propose tokens that a larger model verifies in parallel.
- How it is used
- Accelerates large-model generation when latency matters and additional memory can hold a smaller assistant model.
- How deep it goes
- Explains assistant proposals and parallel verification, then demonstrates model loading without deriving the acceptance algorithm.
- Examples
- Uses Whisper-Large-v3 with Distil-Whisper-Large-v3 on an English audio dataset and provides a Transformers implementation.
- What you can do after
- Apply speculative decoding through Transformers by loading a large model, an assistant model, and generation settings.
Domain-Specific Small Language Models
An inference-time optimization that predicts multiple future tokens in parallel using draft proposals instead of generating strictly one token at a time.
- How it is used
- Uses a draft model to accelerate local LLM generation while a target model preserves the final token decisions.
- How deep it goes
- Explains the two-model verification loop and tokenizer constraint, but stops short of formal acceptance algorithms or performance equations.
- Examples
- Demonstrates LM Studio using Llama 3.2 3B Instruct as target and Llama 3.2 1B Instruct as draft.
- What you can do after
- Operate/configure speculative decoding in LM Studio with compatible draft and target models.
Read how Domain-Specific Small Language Models explains speculative decoding →
Related concepts
- draft modeluses · A draft model is a smaller, faster language model that proposes candidate tokens for verification by a target model during speculative decoding.
- tokenizeruses · A tokenizer is a text-processing component or algorithm that segments text into discrete tokens and may map them to identifiers for computational processing.
- forward passuses · A forward pass is a calculation that propagates inputs through an ordered computational or dependency structure to produce outputs or derived values.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.