liveBook · Manning

Sparsity

As explained in Build a DeepSeek Model (From Scratch)

Sparsity is a computational property in which only a subset of possible model connections, components, or parameters is active for a given computation.

Sparsity activates only a subset of possible model components for each computation. In Mixture-of-Experts (MoE), it replaces one dense feed-forward network with parallel expert networks while keeping most experts dormant for each token. This preserves a large total parameter count without proportional training or inference cost. section 4.1.2 section 4.7

A sparse MoE computation proceeds through three linked decisions:

  • A learnable routing layer scores the available experts for each token. section 4.7
  • Top-k selection activates only the k highest-scoring experts; with k=2, one token uses two of three experts. section 4.2.2
  • A softmax converts selected scores into weights for combining expert outputs into the single matrix expected by the next Transformer layer. section 4.2.1 section 4.7
  • Specialization makes selective activation useful: experts can learn patterns for punctuation, verbs, domain information, or Python code while unrelated experts remain dormant. section 4.1.3 section 4.7

Routing “What is 1+1?”

A router may activate a punctuation expert for the question mark and a verb expert for “is,” while leaving noun and proper-noun experts inactive. The example illustrates how sparsity avoids processing every token through every expert. section 4.1.3

Figure 4.6 The principle of sparsity or load balancing. For each token, we decide to route it to only a subset (k=2) of the available experts.

Shows selective activation visually: each token is routed to only a subset of available experts, such as two of three. section 4.2.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains expert replacement, top-k routing, output weighting, and specialization, but defers detailed load-balancing mechanisms.
Examples
Uses three-expert top-k routing, the question “What is 1+1?”, and specialized punctuation, verb, noun, and code experts.
What it assumes
Contrasts sparse MoE routing with dense feed-forward networks and assumes familiarity with Transformer layers and token processing.
What you can do after
Apply it via top-k MoE routing to increase total parameters without activating every expert for each token.
How it connects
Connects sparsity to Mixture-of-Experts, Transformer feed-forward networks, routing, softmax, expert specialization, and load balancing.

Also explained in

Compare how 6 books explain sparsity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.