liveBook · Manning

Fine-grained expert segmentation

As explained in Build a DeepSeek Model (From Scratch)

Fine-grained expert segmentation is a mixture-of-experts design that replaces a small pool of large experts with a larger pool of smaller, more specialized experts while preserving aggregate capacity.

Fine-grained expert segmentation addresses knowledge hybridity in mixture-of-experts models. Instead of using a few large experts that each become generalists, it routes tokens through a much larger pool of smaller, more specialized experts. section 4.4.2

Capacity-preserving segmentation

A conventional MoE with 16 experts of hidden dimension 4096 has aggregate expert capacity of 16 × 4096. A fine-grained design can use 64 experts of hidden dimension 1024, preserving 64 × 1024 = 16 × 4096 while giving routing a wider specialization pool. Total expert parameters, activated top-k parameters, and inference cost can remain unchanged. section 4.4.2

Figure 4.20 A comparison between a conventional MoE with a few large experts (top) and DeepSeek's fine-grained approach with many smaller experts (bottom).

Compare few large experts with many smaller experts to see how segmentation preserves aggregate capacity while enabling finer specialization. section 4.4.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains expert resizing, aggregate capacity, and activated parameters, but does not detail implementation code or routing algorithms.
Examples
Uses 16 experts of dimension 4096 versus 64 experts of dimension 1024, plus punctuation, verbs, proper nouns, and code as specialization targets.
What you can do after
Evaluate expert-count and expert-size trade-offs while preserving aggregate capacity and activated computation.
How it connects
Connects mixture-of-experts with feed-forward networks, routing mechanisms, top-k selection, and expert specialization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.