Speculative decoding

As explained in Domain-Specific Small Language Models

An inference-time optimization that predicts multiple future tokens in parallel using draft proposals instead of generating strictly one token at a time.

Speculative decoding reduces language-model generation time by pairing a small, fast draft model with a larger target model. The draft model proposes several future tokens in parallel; the target model verifies them and accepts only tokens it would generate itself. Both models must use the same tokenizer, preserving output quality while exploiting parallel Transformer evaluation. section 12.3

Operational requirements and tradeoffs:

• Run two models side by side: a faster draft model and a larger target model. section 12.3
• Use matching tokenizers for the draft and target models. section 12.3
• Expect speedups when the target accepts enough proposed tokens; rejected proposals reduce the benefit. section 12.3
• Configure the feature alongside inference settings such as sampling parameters, context length, seed, and KV-cache quantization. section 12.3

LM Studio pairing

LM Studio demonstrates speculative decoding with Llama 3.2 3B Instruct as the target and Llama 3.2 1B Instruct as the draft model. The feature supports GGUF and Apple MLX models and can serve local requests without an internet or LAN connection after model download. section 12.3 section 12.7

Figure 12.8 Speculative Decoding in action with the Llama 3.2 3B Instruct (target) and Llama 3.2 1B Instruct (draft) models in LM Studio

Shows speculative decoding in action with separate target and draft models in LM Studio. section 12.3

How Domain-Specific Small Language Models covers it

How deep it goes

Explains the two-model verification loop and tokenizer constraint, but stops short of formal acceptance algorithms or performance equations.

Examples

Demonstrates LM Studio using Llama 3.2 3B Instruct as target and Llama 3.2 1B Instruct as draft.

What you can do after

Operate/configure speculative decoding in LM Studio with compatible draft and target models.

How it connects

Connects speculative decoding with Transformer models, tokenizers, local inference, model quantization, and sampling parameters.

liveBookmanning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  LLMs in ProductionSpeculative decoding accelerates generation by having a smaller assistant model propose tokens that a larger model verifies in parallel.

Compare how 2 books explain speculative decoding →

Related concepts

• draft modeluses · A draft model is a smaller, faster language model that proposes candidate tokens for verification by a target model during speculative decoding.
• tokenizeruses · A tokenizer is a text-processing component or algorithm that segments text into discrete tokens and may map them to identifiers for computational processing.
• forward passuses · A forward pass is a calculation that propagates inputs through an ordered computational or dependency structure to produce outputs or derived values.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.