liveBook · Manning

Expert network

As explained in Build a DeepSeek Model (From Scratch)

An expert is a specialized neural subnetwork within a mixture-of-experts model that processes tokens selected by a routing mechanism.

An expert network is a feed-forward network (FFN) within a mixture-of-experts (MoE) layer. Instead of applying one dense FFN to every token, an MoE layer offers a pool of experts and activates only a small subset per token. This separates total model capacity from the computation performed for each token. section 4.1 section 4.1.2

Figure 4.3 The architectural change of MoE. The single, dense FFN is replaced by a collection of four smaller, specialized expert networks.

Compare a single dense FFN with a pool of four expert networks. section 4.1.2

A learnable router scores each expert for each token. Top-k selection chooses the experts to run, and softmax converts their selected scores into weights for combining their outputs. The combined result preserves the input shape expected by the next Transformer layer. section 4.2.3 section 4.7 section 4.2.1

Routing four tokens

For four 8-dimensional token embeddings and three experts, a (4, 8) input multiplied by an (8, 3) routing matrix produces a (4, 3) score matrix. With k=2, each token goes to two experts; for example, “The” may go to E1 and E2 while E3 is skipped. section 4.2.3 section 4.2.2

Experts can learn different patterns during pre-training, such as punctuation or verbs, rather than all performing the same role. But specialization depends on routing: if favored experts receive more tokens, they can improve and attract still more traffic, while rarely selected experts receive few or no updates. section 4.1.3 section 4.3

Two design choices address competing demands on expert capacity:

  • Fine-grained segmentation divides capacity among more, smaller experts. For example, 16 experts with hidden dimension 4096 and 64 with hidden dimension 1024 have the same summed hidden dimensions; activated capacity can also be kept comparable by adjusting top-k. section 4.4.2
  • Shared experts process every token to capture common knowledge, while routed experts handle selected tokens. This reduces the need for multiple specialists to duplicate foundational knowledge. section 4.5 section 4.7

In a PyTorch implementation, an ExpertFFN uses two bias-free linear layers with GELU and dropout between them. The first maps from model dimension to hidden dimension; the second maps back. The full MoE module adds the shared output, routed output, and residual input. section 4.5

Listing 4.1 The ExpertFFN Module
def _gelu(x: torch.Tensor) -> torch.Tensor:
    # Slightly faster GELU (approx)
    return 0.5 * x * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) *
    [CA](x + 0.044715 * torch.pow(x, 3))))
 
class ExpertFFN(nn.Module):
    """
    A 2-layer MLP expert. Hidden dim is usually smaller than a dense FFN
    (e.g., 0.25 × d_model in DeepSeek-V3).
    """
    def __init__(self, d_model: int, hidden: int, dropout: float = 0.0):
        super().__init__()
        self.fc1 = nn.Linear(d_model, hidden, bias=False)
        self.fc2 = nn.Linear(hidden, d_model, bias=False)
        self.dropout = nn.Dropout(dropout)
 
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.fc2(self.dropout(_gelu(self.fc1(x))))

Inspect the two-layer ExpertFFN and its forward pass. section 4.5

Load balancing encourages experts to contribute rather than become hotspots or dead weight. One implementation adjusts a router bias according to observed expert selection counts, without adding an auxiliary balancing loss to the training objective. section 4.3 section 4.5 section 4.7

A controlled comparison

In a 5,000-iteration TinyStories run with nearly equal parameter counts, a DeepSeek-MoE model using shared experts and dynamic balancing reached validation loss 1.9451, versus 1.9854 for a standard MoE. This compares complete designs, not expert segmentation alone. section 4.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Develops expert routing from matrix dimensions and top-k selection through a PyTorch FFN and a complete shared-and-routed forward pass.
Examples
Uses a four-token routing matrix, a 16-versus-64 expert capacity comparison, and a TinyStories model comparison.
What you can do after
Build a PyTorch expert FFN and integrate shared and routed experts into an MoE module.
How it connects
Connects expert networks to feed-forward networks, routing, sparsity, load balancing, and residual connections.

Also explained in

Compare how 5 books explain expert network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.