An expert network is a feed-forward network (FFN) within a mixture-of-experts (MoE) layer. Instead of applying one dense FFN to every token, an MoE layer offers a pool of experts and activates only a small subset per token. This separates total model capacity from the computation performed for each token. section 4.1 section 4.1.2
Figure 4.3 The architectural change of MoE. The single, dense FFN is replaced by a collection of four smaller, specialized expert networks.

Compare a single dense FFN with a pool of four expert networks. section 4.1.2
A learnable router scores each expert for each token. Top-k selection chooses the experts to run, and softmax converts their selected scores into weights for combining their outputs. The combined result preserves the input shape expected by the next Transformer layer. section 4.2.3 section 4.7 section 4.2.1
Routing four tokens
For four 8-dimensional token embeddings and three experts, a (4, 8) input multiplied by an (8, 3) routing matrix produces a (4, 3) score matrix. With k=2, each token goes to two experts; for example, “The” may go to E1 and E2 while E3 is skipped. section 4.2.3 section 4.2.2
Experts can learn different patterns during pre-training, such as punctuation or verbs, rather than all performing the same role. But specialization depends on routing: if favored experts receive more tokens, they can improve and attract still more traffic, while rarely selected experts receive few or no updates. section 4.1.3 section 4.3
Two design choices address competing demands on expert capacity:
- Fine-grained segmentation divides capacity among more, smaller experts. For example, 16 experts with hidden dimension 4096 and 64 with hidden dimension 1024 have the same summed hidden dimensions; activated capacity can also be kept comparable by adjusting top-k. section 4.4.2
- Shared experts process every token to capture common knowledge, while routed experts handle selected tokens. This reduces the need for multiple specialists to duplicate foundational knowledge. section 4.5 section 4.7
In a PyTorch implementation, an ExpertFFN uses two bias-free linear layers with GELU and dropout between them. The first maps from model dimension to hidden dimension; the second maps back. The full MoE module adds the shared output, routed output, and residual input. section 4.5
Listing 4.1 The ExpertFFN Module
def _gelu(x: torch.Tensor) -> torch.Tensor:
# Slightly faster GELU (approx)
return 0.5 * x * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) *
[CA](x + 0.044715 * torch.pow(x, 3))))
class ExpertFFN(nn.Module):
"""
A 2-layer MLP expert. Hidden dim is usually smaller than a dense FFN
(e.g., 0.25 × d_model in DeepSeek-V3).
"""
def __init__(self, d_model: int, hidden: int, dropout: float = 0.0):
super().__init__()
self.fc1 = nn.Linear(d_model, hidden, bias=False)
self.fc2 = nn.Linear(hidden, d_model, bias=False)
self.dropout = nn.Dropout(dropout)
def forward(self, x: torch.Tensor) -> torch.Tensor:
return self.fc2(self.dropout(_gelu(self.fc1(x))))Inspect the two-layer ExpertFFN and its forward pass. section 4.5
Load balancing encourages experts to contribute rather than become hotspots or dead weight. One implementation adjusts a router bias according to observed expert selection counts, without adding an auxiliary balancing loss to the training objective. section 4.3 section 4.5 section 4.7
A controlled comparison
In a 5,000-iteration TinyStories run with nearly equal parameter counts, a DeepSeek-MoE model using shared experts and dynamic balancing reached validation loss 1.9451, versus 1.9854 for a standard MoE. This compares complete designs, not expert segmentation alone. section 4.6




