liveBook · Manning

Target model

As explained in Rearchitecting LLMs

A target model verifies candidate tokens proposed by a draft model during speculative decoding.

In speculative decoding, the target model verifies candidate tokens proposed autoregressively by a faster draft model. It evaluates the candidates together in one forward pass, accepts matching tokens, and supplies a correction at the first disagreement. If all candidates match, it contributes one additional token. section 13.6.1

Under greedy decoding, verification preserves the target model’s output, aside from rare fp16 numerical ties. Speed depends on draft-step cost and the fraction of candidates accepted.

  • Compare target-only and speculative generation under the same conditions; the Hugging Face workflow uses do_sample=False and measures elapsed time. section 13.6.1
  • Formulaic agent outputs can favor speculation: draft models may handle tags, JSON keys, whitespace, and known tool names, while the target checks precise argument values. section 13.6.1
Figure 13.5 One iteration of speculative decoding with K = 5. The draft model S proposes five candidate tokens one at a time; the target model T verifies all five in a single forward pass. Tokens 1–3 are accepted. The first disagreement occurs at token 4, so tokens 4 and 5 are discarded and T writes one token of its own in position 4. In the next iteration, S resumes drafting from T's token. The tokens are illustrative digits of a ZIP code, and the disagreement at token 4 is constructed for this example.

The five-token illustration shows acceptance through token 3, correction at token 4, and discarded draft tokens 4 and 5. section 13.6.1

How Rearchitecting LLMs covers it

How deep it goes
Explains batched verification, acceptance, correction, and greedy equivalence, while noting rare fp16 numerical ties.
Examples
Uses a five-token ZIP-code illustration and formulaic agent outputs with tags, JSON keys, whitespace, and tool names.
What it assumes
Assumes paired draft and target models, and defines greedy decoding as selecting the most probable token at each step.
What you can do after
Operate Hugging Face generation with an assistant model and compare speculative output and latency against target-only generation.
How it connects
Connects speculative decoding to draft models, greedy decoding, and token acceptance.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.