In speculative decoding, the target model verifies candidate tokens proposed autoregressively by a faster draft model. It evaluates the candidates together in one forward pass, accepts matching tokens, and supplies a correction at the first disagreement. If all candidates match, it contributes one additional token. section 13.6.1
Under greedy decoding, verification preserves the target model’s output, aside from rare fp16 numerical ties. Speed depends on draft-step cost and the fraction of candidates accepted.
- Compare target-only and speculative generation under the same conditions; the Hugging Face workflow uses do_sample=False and measures elapsed time. section 13.6.1
- Formulaic agent outputs can favor speculation: draft models may handle tags, JSON keys, whitespace, and known tool names, while the target checks precise argument values. section 13.6.1
Figure 13.5 One iteration of speculative decoding with K = 5. The draft model S proposes five candidate tokens one at a time; the target model T verifies all five in a single forward pass. Tokens 1–3 are accepted. The first disagreement occurs at token 4, so tokens 4 and 5 are discarded and T writes one token of its own in position 4. In the next iteration, S resumes drafting from T's token. The tokens are illustrative digits of a ZIP code, and the disagreement at token 4 is constructed for this example.

The five-token illustration shows acceptance through token 3, correction at token 4, and discarded draft tokens 4 and 5. section 13.6.1
