liveBook · Manning

Draft model

As explained in Rearchitecting LLMs

A draft model proposes candidate tokens for a target model to verify during speculative decoding.

A draft model proposes candidate tokens autoregressively; a target model verifies them together in one forward pass. The target accepts matching tokens, corrects the first disagreement, and drafting resumes from there. section 13.6.1

Predictable agent output

Tags, JSON keys, whitespace, and known tool names are predictable structural tokens a draft model can propose; the target can focus its corrections on argument values. section 13.6.1

Figure 13.5 One iteration of speculative decoding with K = 5. The draft model S proposes five candidate tokens one at a time; the target model T verifies all five in a single forward pass. Tokens 1–3 are accepted. The first disagreement occurs at token 4, so tokens 4 and 5 are discarded and T writes one token of its own in position 4. In the next iteration, S resumes drafting from T's token. The tokens are illustrative digits of a ZIP code, and the disagreement at token 4 is constructed for this example.

Follow a five-token ZIP-code example: the target accepts three tokens, replaces the fourth after a disagreement, and the draft resumes from that correction. section 13.6.1

How Rearchitecting LLMs covers it

How deep it goes
Explains the propose–verify loop and greedy-output equivalence, but does not detail model internals.
Examples
Uses predictable agent syntax and a five-token ZIP-code illustration to show drafting and correction.
What you can do after
Apply speculative decoding by pairing a draft model with a target model that verifies candidate tokens.
How it connects
Pairs with target models in speculative decoding and uses greedy decoding to preserve the target’s output.

Also explained in

Compare how 2 books explain draft model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.