liveBook · Manning

Top-k gating

As explained in Build a DeepSeek Model (From Scratch)

Top-k gating selects the highest-scoring experts for each token and assigns normalized routing weights to them.

Top-k gating routes each token to a small, specialized subset of experts. A router scores every expert, adds a dynamic balancing bias, selects the highest-scoring k experts, and normalizes their scores with softmax. section 4.5

An implementation proceeds through four tensor operations:

  • Dispatch tokens by expert using a boolean mask and nonzero indices, process each batch through its expert, and add weighted outputs back with index_add_. section 4.5
Figure 4.27 The complete forward pass of the DeepSeekMoE module, showing the three parallel data paths: the dense shared expert path, the sparse routed expert path with dynamic load balancing, and the residual connection.

Trace top-k gating within the complete DeepSeekMoE flow, alongside shared experts, routed experts, and the residual connection. section 4.5

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Implements logits, biasing, top-k selection, dispatch, weighted recombination, and gradient-free load balancing.
Examples
Uses PyTorch listings for router computation, torch.topk selection, masked dispatch, indexed accumulation, and bias updates.
What it assumes
Assumes familiarity with tensors, linear projections, softmax, expert modules, and indexed PyTorch operations.
What you can do after
Build it from scratch as a sparse PyTorch routing path with normalized gates and dynamic expert-load correction.
How it connects
Connects mixture-of-experts routing with softmax, sparse dispatch, shared experts, residual connections, and load balancing.

Also explained in

Compare how 2 books explain top-k gating →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.