Dynamic inference makes runtime decisions about which computations to perform, tokens to process, or experts to activate. It can reduce work compared with applying the full model structure to every token, but requires input-dependent execution decisions. section 9.1
Three approaches make different runtime decisions:
- Early exiting adds intermediate exits; logits indicate confidence, and the model stops when confidence passes a threshold. section 9.1
- Token dropping uses attention weights to assess token importance and discards tokens considered dispensable. section 9.1
- Mixture of Experts routes each token to selected experts. A trained router uses contextual hidden states, and expert outputs are combined. section 9.6
Table 9.1 Dynamic inference techniques
|
Technique
|
What it decides
|
Decision mechanism
|
Where it acts
|
Main objective
|
|
Early Exiting
|
When to stop
|
Logit concentration on a single token
|
Model blocks
|
Reduce the number of processed blocks
|
|
Token Dropping
|
Which tokens to process
|
Attention weights
|
KV cache
|
Reduce the active sequence length
|
|
Mixture of Experts
|
Which expert to route to
|
Trained linear router
|
MLP module
|
Direct each token to the most suitable expert
|
Compare what each technique decides, its decision mechanism, where it acts, and its main objective. section 9.1
