liveBook · Manning

Expert network

An expert network is a specialized neural subnetwork within a mixture-of-experts model that processes inputs selected or routed to it.

How 5 Manning books explain it

Rearchitecting LLMs

An expert is an independently parameterized neural-network component in a mixture-of-experts architecture that processes selected inputs.

How it is used
Uses expert networks to add specialized processing to a general model while routing tokens dynamically.
How deep it goes
Implements expert initialization, freezing, routing, training, transplantation, and evaluation, including top-k inference and compatibility checks.
Examples
Uses general and clinical MLP experts, a transplanted clinical expert, and a three-expert general, clinical, and Python-code exercise.
What you can do after
Builds expert-based MoE blocks from existing MLPs, trains selected experts and routers, and inspects routing alongside task performance.

Read how Rearchitecting LLMs explains expert network →

Build a DeepSeek Model (From Scratch)

An expert is a specialized neural subnetwork within a mixture-of-experts model that processes tokens selected by a routing mechanism.

How it is used
Replaces a dense Transformer FFN with selectively activated subnetworks to increase capacity without proportional per-token computation.
How deep it goes
Develops expert routing from matrix dimensions and top-k selection through a PyTorch FFN and a complete shared-and-routed forward pass.
Examples
Uses a four-token routing matrix, a 16-versus-64 expert capacity comparison, and a TinyStories model comparison.
What you can do after
Build a PyTorch expert FFN and integrate shared and routed experts into an MoE module.

Read how Build a DeepSeek Model (From Scratch) explains expert network →

CUDA for LLMs

An expert is a subnetwork in a mixture-of-experts model whose learned weights specialize in processing particular inputs.

How it is used
Uses sparse expert routing in transformer inference to reduce per-token computation while retaining a larger total model capacity.
How deep it goes
Implements gating, softmax, top-k selection, score normalization, and expert execution, without detailing production-optimized routing.
Examples
Uses an inference model with eight experts, top-2 routing, and roughly 963 million total parameters.
What you can do after
Build a PyTorch inference layer that routes tokens to selected experts and combines their weighted outputs.

Read how CUDA for LLMs explains expert network →

Transformers in Action

An expert is a specialized neural subnetwork within a mixture-of-experts architecture that processes inputs routed to it.

How it is used
Scales transformer capacity by routing each token through only a small subset of specialized feed-forward subnetworks.
How deep it goes
Explains expert placement, routing intuition, top-k selection, and implementation variants, but stops short of full training mechanics.
Examples
Uses top-k gating code and GShard-style top-two routing with capacity, load balancing, and random-routing options.
What you can do after
Configure sparse expert routing with top-k selection and recognize its computational trade-offs against dense transformer layers.

Read how Transformers in Action explains expert network →

Machines that Think

A specialized subnetwork within a mixture-of-experts layer that processes inputs when selected by the gating mechanism.

How it is used
Provides specialized computation for selected token representations inside a mixture-of-experts layer.
How deep it goes
Defines expert selection and weighted combination, then traces specialization through training feedback.
Examples
Illustrates routing with a family-doctor analogy and cites Mixtral and, reportedly, GPT-4.
What you can do after
Understand how selective activation expands capacity while bounding per-token computation, and why routing creates workload trade-offs.

Read how Machines that Think explains expert network →

Also covered in

What to know first

Related concepts

Questions about expert network

What is expert network?

An expert network is a specialized neural subnetwork within a mixture-of-experts model that processes inputs selected or routed to it.

Which Manning books explain expert network?

Rearchitecting LLMs, Build a DeepSeek Model (From Scratch), CUDA for LLMs, Transformers in Action and Machines that Think.

What should I know before learning expert network?

Manning books that teach expert network usually introduce feedforward network, token, linear layer and transformer architecture first.

What is expert network a kind of?

Expert network is a kind of feedforward network. A feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback.

What is expert network part of?

Expert network is part of mixture of experts. A mixture-of-experts model is a neural-network architecture with specialized subnetworks and a routing mechanism that selects or combines experts for each input.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.