liveBook · Manning

Sparse dispatch

As explained in Build a DeepSeek Model (From Scratch)

Sparse dispatch batches tokens by assigned expert, computes expert outputs, and combines the outputs at their original token positions.

Sparse dispatch routes each token to a small top-k subset of experts, batches tokens selected by the same expert, and restores results to their original positions. Router logits receive a balancing bias; top-k scores become normalized gates. section 4.5

An implementation follows four tensor operations:

  • Compute router logits and add the dynamic expert-load bias. section 4.5
  • Select top-k expert indices and calculate softmax gate weights. section 4.5
  • For each expert, mask selected token rows, gather them with nonzero indices, and process the resulting batch. section 4.5
  • Multiply expert outputs by their gates and use index_add_ to accumulate them at the original token positions. section 4.5

Routed-path accumulation

With topk_idx == i, the boolean mask identifies every token assigned to expert i. row_idx selects those tokens, which_k selects their gate values, and index_add_ writes weighted outputs into routed_out. Shared-expert output and the routed result then join the final MoE output. section 4.5

Figure 4.27 The complete forward pass of the DeepSeekMoE module, showing the three parallel data paths: the dense shared expert path, the sparse routed expert path with dynamic load balancing, and the residual connection.

Use the forward-pass diagram to trace the shared path, sparse routed path, dynamic load balancing, and residual combination together. section 4.5

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains routing through batching, masking, indexed gathering, gated computation, and indexed accumulation, while deferring kernel-level optimization.
Examples
Uses PyTorch masks, nonzero(), index_select(), and index_add_() to implement per-expert routing and recombination.
What it assumes
Assumes tensor indexing, linear layers, top-k selection, softmax gating, and basic mixture-of-experts structure.
What you can do after
Builds a sparse routed expert path that batches assignments and recombines outputs at token positions.
How it connects
Connects top-k gating, softmax, shared experts, residual connections, and auxiliary-loss-free load balancing.

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.