Speculative decoding accelerates generation by having a smaller draft model propose tokens for a larger target model to verify. In an agent-model family, the draft can handle predictable structure while the target preserves its greedy output. section 13.6 section 13.6.1
A generation iteration follows this loop:
- Have the draft model propose K tokens autoregressively. section 13.6.1
- Have the target evaluate all K tokens in one forward pass; accept agreed tokens and replace the first disagreement with a target-generated token. section 13.6.1
- Resume drafting from the accepted prefix or corrected token. If all proposals are accepted, the target still adds one token. section 13.6.1
A tool-call generation
With K = 5, the target accepts three illustrative ZIP-code digits, rejects the fourth proposal, and supplies its own token before drafting resumes. In the notebook workflow, generation uses do_sample=False and compares target-only output with output using S as the assistant model. section 13.6.1 section 13.6.1
Figure 13.5 One iteration of speculative decoding with K = 5. The draft model S proposes five candidate tokens one at a time; the target model T verifies all five in a single forward pass. Tokens 1–3 are accepted. The first disagreement occurs at token 4, so tokens 4 and 5 are discarded and T writes one token of its own in position 4. In the next iteration, S resumes drafting from T's token. The tokens are illustrative digits of a ZIP code, and the disagreement at token 4 is constructed for this example.

Shows one K = 5 iteration, including accepted proposals, the first disagreement, target correction, and where drafting resumes. section 13.6.1
Under greedy decoding, verification matches the target’s output, apart from rare fp16 numerical ties. Speed depends on draft acceptance and the relative cost of draft and target steps; predictable tool-call syntax can improve acceptance. section 13.6.1


