liveBook · Manning

Speculative decoding

As explained in Rearchitecting LLMs

Speculative decoding is a generation technique in which a smaller model proposes tokens that a larger model verifies, preserving the larger model’s output while potentially accelerating generation.

Speculative decoding accelerates generation by having a smaller draft model propose tokens for a larger target model to verify. In an agent-model family, the draft can handle predictable structure while the target preserves its greedy output. section 13.6 section 13.6.1

A generation iteration follows this loop:

  • Have the draft model propose K tokens autoregressively. section 13.6.1
  • Have the target evaluate all K tokens in one forward pass; accept agreed tokens and replace the first disagreement with a target-generated token. section 13.6.1
  • Resume drafting from the accepted prefix or corrected token. If all proposals are accepted, the target still adds one token. section 13.6.1

A tool-call generation

With K = 5, the target accepts three illustrative ZIP-code digits, rejects the fourth proposal, and supplies its own token before drafting resumes. In the notebook workflow, generation uses do_sample=False and compares target-only output with output using S as the assistant model. section 13.6.1 section 13.6.1

Figure 13.5 One iteration of speculative decoding with K = 5. The draft model S proposes five candidate tokens one at a time; the target model T verifies all five in a single forward pass. Tokens 1–3 are accepted. The first disagreement occurs at token 4, so tokens 4 and 5 are discarded and T writes one token of its own in position 4. In the next iteration, S resumes drafting from T's token. The tokens are illustrative digits of a ZIP code, and the disagreement at token 4 is constructed for this example.

Shows one K = 5 iteration, including accepted proposals, the first disagreement, target correction, and where drafting resumes. section 13.6.1

Under greedy decoding, verification matches the target’s output, apart from rare fp16 numerical ties. Speed depends on draft acceptance and the relative cost of draft and target steps; predictable tool-call syntax can improve acceptance. section 13.6.1

How Rearchitecting LLMs covers it

How deep it goes
Explains draft proposals, parallel target verification, correction, and greedy-output guarantees, while noting rare fp16 ties.
Examples
Uses a five-token ZIP-code illustration and compares target-only generation with a speculative tool-call output.
What it assumes
Assumes draft and target models share a tokenizer, vocabulary, and base lineage.
What you can do after
Apply Hugging Face generation with an assistant model to compare speculative output and timing against target-only generation.
How it connects
Connects draft and target models with greedy decoding, knowledge distillation, and cascaded routing.

Also explained in

Compare how 3 books explain speculative decoding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.