When choosing a model to self-host, parameter count is only part of the capacity decision. The model’s architecture also affects how much memory and computation it needs for each token. section 1.1.8
A dense model uses all of its parameters—the learned numbers that make up the model—for every token it processes. In a transformer, each token passes through the whole feed-forward layer of every block. Attention lets a token take other tokens into account; the feed-forward layer then processes each token on its own. section 1.1.8
A Mixture of Experts (MoE) model instead splits feed-forward layers into experts and routes each token to a few of them. Its attention layers remain dense. All experts must still be available in memory, even though only the selected experts perform most of the computation. section 1.1.8
Figure 1.3 In a dense model, every token passes through the entire feed-forward layer of each transformer block. In an MoE model, attention remains dense, but in MoE transformer blocks the feed-forward layer is replaced by a set of experts. A router selects a small subset of them for each token (here, E1 and E5), and their outputs are combined. This allows the model to scale to many more parameters without activating all of them for every token. The experts typically need to reside in accelerator memory, possibly distributed across multiple GPUs.

Compare the full feed-forward path in a dense transformer block with routed experts in an MoE block. section 1.1.8
Memory versus computation
A 70-billion-parameter dense model stores all 70 billion parameters and activates the entire network for every token. A 200-billion-parameter MoE model stores all 200 billion but might activate only 20 billion per token. The MoE model needs more memory, while its per-token computation may be closer to that of a smaller dense model. Keeping experts in CPU memory or on disk can reduce GPU memory pressure, but at a large speed cost. section 1.1.8
Dense models appear alongside MoE models in practical model families. For example, Mistral publishes both designs, while Muse Glimmer is a 30-billion-parameter dense model released under Apache 2.0. Check licensing as well as capacity when selecting weights for deployment. section 1.1.7

