liveBook · Manning

Dynamic inference

As explained in Rearchitecting LLMs

Dynamic inference is model execution that makes input-dependent runtime decisions about which computations, tokens, or submodels to use.

Dynamic inference makes runtime decisions about which computations to perform, tokens to process, or experts to activate. It can reduce work compared with applying the full model structure to every token, but requires input-dependent execution decisions. section 9.1

Three approaches make different runtime decisions:

  • Early exiting adds intermediate exits; logits indicate confidence, and the model stops when confidence passes a threshold. section 9.1
  • Token dropping uses attention weights to assess token importance and discards tokens considered dispensable. section 9.1
  • Mixture of Experts routes each token to selected experts. A trained router uses contextual hidden states, and expert outputs are combined. section 9.6
Table 9.1 Dynamic inference techniques
Technique
What it decides
Decision mechanism
Where it acts
Main objective
Early Exiting
When to stop
Logit concentration on a single token
Model blocks
Reduce the number of processed blocks
Token Dropping
Which tokens to process
Attention weights
KV cache
Reduce the active sequence length
Mixture of Experts
Which expert to route to
Trained linear router
MLP module
Direct each token to the most suitable expert

Compare what each technique decides, its decision mechanism, where it acts, and its main objective. section 9.1

How Rearchitecting LLMs covers it

How deep it goes
Defines three approaches and their decision mechanisms, but gives limited implementation detail beyond early-exit confidence and token importance.
Examples
Uses confidence-based exits, attention-based token removal, and per-token expert routing as concrete mechanisms.
What you can do after
Understand how runtime choices can reduce computation by stopping early, dropping tokens, or routing to experts.
How it connects
Connects early exiting and token dropping with Mixture of Experts, logits, attention weights, and routers.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.