Mixture of experts

As explained in LLMs in Production

A model composed of multiple specialized expert subnetworks whose participation is selected or weighted for each input.

A mixture of experts (MoE) avoids running every part of a model for every input. In a transformer, an MoE layer can replace a parameter-dense feed-forward layer with multiple expert networks. A gate or router directs an input to the experts selected for it, so only some experts typically run. section 3.3.1

Figure 3.3 Example mixture of an expert's model with both a gate and router to control flow. The MoE model is used to replace the FFN layers in a transformer; here, we show it replacing the FFN in a decoder.

The diagram locates the MoE layer in a transformer decoder and shows how a gate and router control the path through its experts. section 3.3.1

Think of the experts as students who begin together and develop different specialties during training. Their specialties need not have recognizable names: they emerge from patterns in the training data. At inference time, selecting experts relevant to an input can retain that specialization without consulting every expert. Unlike an ensemble that combines every model’s result, an MoE typically runs only one or a few experts; a complex input may call for more. section 5.3.4 section 3.3.1

The practical trade-offs are:

• Sparse activation shortens the computation path, but the full layer can still contain many parameters. Do not equate fewer active experts with fewer stored parameters. section 3.3.1
• Changing a feed-forward layer into an MoE changes the model’s structure and requires finetuning afterward. Finetuning must account for both experts and gating. section 3.3.1 section 5.5
• Quantization can complement MoE: a cited study reached 2-bit quantization with minimal reported accuracy impact. That result is not a guarantee for every model. section 3.3.1

Finetune an existing MoE

Rather than converting a feed-forward layer yourself, load a pretrained Google Switch Transformer through Hugging Face Transformers, tokenize a text dataset, and finetune it. The example uses `google/switch-base-8` and a local text file; it illustrates using an existing MoE rather than building its routing mechanism from scratch. section 5.3.4

How LLMs in Production covers it

How deep it goes

Explains transformer-layer replacement, routing and finetuning implications, but does not specify a router implementation.

Examples

Shows a gate-and-router decoder diagram and finetuning of a pretrained Switch Transformer on a local text dataset.

What it assumes

Assumes familiarity with transformers and finetuning; introduces expert selection through a plain-language specialization analogy.

What you can do after

Apply it via Hugging Face Transformers by finetuning a pretrained Switch Transformer, or evaluate sparse routing as an architectural trade-off.

How it connects

Connects mixture of experts to transformer feed-forward layers, ensembles, quantization and finetuning.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionA sparse mixture of experts is a mixture-of-experts architecture in which a router activates only a subset of expert modules for each input.
• 

  liveBookmanning.com
  Transformers in ActionA mixture-of-experts model routes inputs among specialized subnetworks called experts, typically activating or combining only a selected subset for each input.
• 

  liveBookmanning.com
  Build Applications with Local AI Models on a MacA neural network architecture that routes inputs through selected expert subnetworks rather than activating every parameter for every input.

Compare how 4 books explain mixture of experts →

Related concepts

• expert networkhas part · An expert network is a specialized neural subnetwork within a mixture-of-experts model that processes inputs selected or routed to it by a gating mechanism.
• routinghas part · Routing is the selection of a path, destination, or processing component to direct an item, request, or flow toward an intended endpoint.
• feedforward networkalternative to · A feedforward network is a neural network or subnetwork in which information flows from inputs toward outputs through successive computations without recurrent connections or cycles.
• dense modelcontrasts with · A dense model is a neural model that applies the same computational pathway, rather than selectively routing each input to a subset of components.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.