Expert network

As explained in Transformers in Action

An expert is a specialized neural subnetwork within a mixture-of-experts architecture that processes inputs routed to it.

An expert is a specialized neural subnetwork within a mixture-of-experts architecture. Different experts can specialize in distinct tasks or input characteristics, while routing activates only the relevant subset for each input. section 3.5

In a transformer, expert networks usually replace the feed-forward network after self-attention. Instead of one large FFN processing every token, multiple smaller expert FFNs share the layer’s workload. section 3.5.1

An MoE layer combines:

• A set of expert networks, commonly implemented as separate feed-forward subnetworks. section 3.5.1
• A gating network, or router, that scores experts from each token’s hidden representation. section 3.5.1
• Sparse activation, typically selecting the top one or two experts per token to reduce computation while retaining a larger parameter pool. section 3.5 section 3.5.1

Top-k routing

A sparse gate applies a learned linear transformation to token representations, selects the highest-scoring experts with top-k, and normalizes their selected scores with softmax. A GShard-style variant extends this pattern with top-two routing, capacity controls, load balancing, and optional random routing.

Figure 3.4 Simplified MoE architecture. Only a few experts are active per input, making inference more efficient.

Use the architecture figure to trace how a token is routed to only a few active experts. section 3.5.1

How Transformers in Action covers it

How deep it goes

Explains expert placement, routing intuition, top-k selection, and implementation variants, but stops short of full training mechanics.

Examples

Uses top-k gating code and GShard-style top-two routing with capacity, load balancing, and random-routing options.

What it assumes

Assumes familiarity with transformer layers, token hidden representations, feed-forward networks, and softmax-based scoring.

What you can do after

Configure sparse expert routing with top-k selection and recognize its computational trade-offs against dense transformer layers.

How it connects

Connects mixture-of-experts with transformer blocks, feed-forward networks, self-attention, gating networks, and conditional computation.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Related concepts

• mixture of expertspart of · A mixture-of-experts model is a neural-network architecture with specialized expert subnetworks and a routing mechanism that selects or combines experts for each input.
• routingused by · Routing is the selection of a path, destination, or processing component to direct an item, request, or flow toward an intended endpoint.
• feedforward networkkind of · A feedforward network is a neural network or subnetwork in which information flows from inputs toward outputs through successive computations without recurrent connections or cycles.
• freezeconfigured by · Freezing is the practice of keeping selected model parameters unchanged during training or adaptation, typically while allowing other parameters to be updated.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.