Expert network
An expert network is a specialized neural subnetwork within a mixture-of-experts model that processes inputs selected or routed to it.
Rearchitecting LLMs
An expert is an independently parameterized neural-network component in a mixture-of-experts architecture that processes selected inputs.
- How it is used
- Uses expert networks to add specialized processing to a general model while routing tokens dynamically.
- How deep it goes
- Implements expert initialization, freezing, routing, training, transplantation, and evaluation, including top-k inference and compatibility checks.
- Examples
- Uses general and clinical MLP experts, a transplanted clinical expert, and a three-expert general, clinical, and Python-code exercise.
- What you can do after
- Builds expert-based MoE blocks from existing MLPs, trains selected experts and routers, and inspects routing alongside task performance.
Build a DeepSeek Model (From Scratch)
An expert is a specialized neural subnetwork within a mixture-of-experts model that processes tokens selected by a routing mechanism.
- How it is used
- Replaces a dense Transformer FFN with selectively activated subnetworks to increase capacity without proportional per-token computation.
- How deep it goes
- Develops expert routing from matrix dimensions and top-k selection through a PyTorch FFN and a complete shared-and-routed forward pass.
- Examples
- Uses a four-token routing matrix, a 16-versus-64 expert capacity comparison, and a TinyStories model comparison.
- What you can do after
- Build a PyTorch expert FFN and integrate shared and routed experts into an MoE module.
Read how Build a DeepSeek Model (From Scratch) explains expert network →
CUDA for LLMs
An expert is a subnetwork in a mixture-of-experts model whose learned weights specialize in processing particular inputs.
- How it is used
- Uses sparse expert routing in transformer inference to reduce per-token computation while retaining a larger total model capacity.
- How deep it goes
- Implements gating, softmax, top-k selection, score normalization, and expert execution, without detailing production-optimized routing.
- Examples
- Uses an inference model with eight experts, top-2 routing, and roughly 963 million total parameters.
- What you can do after
- Build a PyTorch inference layer that routes tokens to selected experts and combines their weighted outputs.
Transformers in Action
An expert is a specialized neural subnetwork within a mixture-of-experts architecture that processes inputs routed to it.
- How it is used
- Scales transformer capacity by routing each token through only a small subset of specialized feed-forward subnetworks.
- How deep it goes
- Explains expert placement, routing intuition, top-k selection, and implementation variants, but stops short of full training mechanics.
- Examples
- Uses top-k gating code and GShard-style top-two routing with capacity, load balancing, and random-routing options.
- What you can do after
- Configure sparse expert routing with top-k selection and recognize its computational trade-offs against dense transformer layers.
Machines that Think
A specialized subnetwork within a mixture-of-experts layer that processes inputs when selected by the gating mechanism.
- How it is used
- Provides specialized computation for selected token representations inside a mixture-of-experts layer.
- How deep it goes
- Defines expert selection and weighted combination, then traces specialization through training feedback.
- Examples
- Illustrates routing with a family-doctor analogy and cites Mixtral and, reportedly, GPT-4.
- What you can do after
- Understand how selective activation expands capacity while bounding per-token computation, and why routing creates workload trade-offs.
Also covered in
- Self-Hosting LLMs at ScaleIn early access (MEAP)
- LLMs in Production
- Introduction to Generative AI, Second Edition
- Deep Learning with PyTorch, Second Edition
- Grokking AI Algorithms, Second Edition
What to know first
- feedforward networkA feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
- linear layerA linear layer is a neural-network layer that applies a learned affine transformation to its input, typically using weights and a bias.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
Related concepts
- mixture of expertspart of · A mixture-of-experts model is a neural-network architecture with specialized subnetworks and a routing mechanism that selects or combines experts for each input.
- routingused by · Routing is the process of directing an item, request, or task to a destination or processing path according to relevant rules or characteristics.
- feedforward networkkind of · A feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback.
- routerused by · A router is a component that directs an incoming request or task to a destination or processing path according to routing criteria.
Questions about expert network
What is expert network?
An expert network is a specialized neural subnetwork within a mixture-of-experts model that processes inputs selected or routed to it.
Which Manning books explain expert network?
Rearchitecting LLMs, Build a DeepSeek Model (From Scratch), CUDA for LLMs, Transformers in Action and Machines that Think.
What should I know before learning expert network?
Manning books that teach expert network usually introduce feedforward network, token, linear layer and transformer architecture first.
What is expert network a kind of?
Expert network is a kind of feedforward network. A feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback.
What is expert network part of?
Expert network is part of mixture of experts. A mixture-of-experts model is a neural-network architecture with specialized subnetworks and a routing mechanism that selects or combines experts for each input.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




