liveBook · Manning

Top-k

As explained in Build a DeepSeek Model (From Scratch)

Top-k selection retains the k highest-scoring experts for each token and discards the others.

Top-k selection makes Mixture-of-Experts computation sparse: each token activates only its k highest-scoring experts, leaving the remaining experts dormant. With three experts and k=2, every token uses two expert paths instead of all three. section 4.2.2

Implement top-k routing as a score-filtering pipeline:

  • Generate one raw score for every token–expert pair with the learned routing mechanism. section 4.2.3
  • Select the k largest scores in each token row and discard the rest. section 4.2.4
  • Replace discarded scores with negative infinity so softmax assigns them zero weight. section 4.2.4
  • Apply softmax to the masked row, producing nonzero expert weights that sum to one. section 4.2.4

Three experts, k=2

For the token "The," suppose the router scores E1, E2, and E3 as 1, 5, and 4. Keep E2 and E3, mask E1, and normalize the survivors; the resulting weights can be 60% for E2 and 40% for E3. section 4.2.4

Figure 4.8 The top-k selection process. For each row, only the two highest scores are kept, and the rest are masked out.

Visualize row-wise top-k selection before masking the discarded experts. section 4.2.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains row-wise selection, negative-infinity masking, and softmax weighting, but stops short of implementing expert-output merging.
Examples
Uses three experts, k=2, the token "The," scores 1, 5, 4, and 60%/40% normalized weights.
What it assumes
Assumes tokens, experts, router scores, and softmax, while illustrating the selection pipeline from raw scores.
What you can do after
Apply it via a router to select experts and compute normalized weights for each token.
How it connects
Connects routing mechanisms, expert selector scores, softmax normalization, sparse activation, and load balancing.

Also explained in

Compare how 3 books explain top-k →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.