When sizing infrastructure for a language model, parameter count alone does not describe the work done for each token. A mixture of experts (MoE) model divides its feed-forward layers into smaller components called experts. A router selects only a few experts for each token, and their outputs are combined. Attention remains dense. section 1.1.8
Figure 1.3 In a dense model, every token passes through the entire feed-forward layer of each transformer block. In an MoE model, attention remains dense, but in MoE transformer blocks the feed-forward layer is replaced by a set of experts. A router selects a small subset of them for each token (here, E1 and E5), and their outputs are combined. This allows the model to scale to many more parameters without activating all of them for every token. The experts typically need to reside in accelerator memory, possibly distributed across multiple GPUs.

The diagram contrasts dense feed-forward processing with routing to selected experts. section 1.1.8
The distinction matters for deployment: all experts must remain available in memory, even though only selected experts perform most of the computation for a token. Experts typically reside in accelerator memory and may be distributed across GPUs. Some tools can place them in CPU memory or on disk, but at a large speed cost. section 1.1.8 section 1.8
Sizing by stored and active parameters
A 200-billion-parameter MoE might activate only 20 billion parameters per token. It still needs all 200 billion available, but its computation per token may be closer to that of a much smaller dense model. Plan for the full memory requirement rather than sizing from the active parameter count alone. section 1.1.8









