liveBook · Manning

Draft model

As explained in Domain-Specific Small Language Models

The smaller, faster model in speculative decoding that proposes candidate tokens for verification by a target model.

In speculative decoding, the draft model is the smaller, faster model that proposes candidate tokens for a larger target model to verify. The pair works side by side and must share a tokenizer. section 12.3 section 12.7

The draft model is chosen to run quickly and generate several next tokens in advance. The target model then accepts or rejects each candidate, keeping only tokens it would have produced itself. section 12.3 section 12.3

Key constraints shape the setup:

  • It should be much smaller than the target model. section 12.3
  • It should come from the same family as the target model. section 12.3
  • It must use the same tokenizer as the target model. section 12.3

LM Studio speculative decoding example

A 3B Instruct model is used as the target and a 1B Instruct model as the draft, both in 4-bit quantized form. The speculative decoding panel can visualize accepted draft tokens during generation. section 12.3 section 12.3

How Domain-Specific Small Language Models covers it

How deep it goes
Defines the role and operating constraints, then stops short of model construction or training details.
Examples
Uses LM Studio with Llama 3.2 3B Instruct as target and Llama 3.2 1B Instruct as draft.
What it assumes
Assumes only a larger target model and a smaller draft model; it explicitly requires a shared tokenizer.
What you can do after
Apply it to speed up generation when enough proposed tokens are accepted.
How it connects
Ties speculative decoding to target model, tokenizer, Transformer models, and LM Studio.
Cover of Domain-Specific Small Language Models
liveBookmanning.com
Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.