liveBook · Manning

Expert-choice routing

As explained in Transformers in Action

Expert-choice routing is a routing strategy in which each expert selects the tokens it will process rather than receiving assignments from a token-side gate.

Expert-choice routing reverses conventional sparse gating: each expert selects the tokens it will process instead of receiving token assignments from a gate. section 3.5.1

This inversion improves load balancing by construction and can reduce expert collapse, but it may leave some tokens unselected, creating coverage gaps. section 3.5.1

How Transformers in Action covers it

How deep it goes
Defines the routing inversion and its trade-offs, but stops short of detailing an expert-choice implementation.
Examples
Contrasts expert-side selection with sparse top-k gating and mentions auxiliary losses as an alternative balancing technique.
What it assumes
Assumes familiarity with mixture-of-experts models, gating, tokens, and expert specialization.
What you can do after
Evaluate routing designs by balancing expert utilization against the risk of leaving tokens unprocessed.
How it connects
Connects sparse gating, soft gating, hard gating, dense gating, load balancing, auxiliary losses, and expert collapse.
Cover of Transformers in Action
liveBookmanning.com
Read Transformers in Action on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.