liveBook · Manning

Routed path

As explained in Build a DeepSeek Model (From Scratch)

A routed path is the sparse MoE computation in which each token is processed by a selected subset of experts.

A routed path is the sparse branch of a mixture-of-experts layer: it scores each token against routed experts, selects top-k specialists, and processes tokens only through those assignments. section 4.5

A practical routed path proceeds by:

  • Computing router logits from flattened token representations and expert centroids, then adding a dynamic load-balancing bias. section 4.5
  • Dispatching tokens with expert-specific indexing, combining expert outputs, and adding them to the shared path and residual input. section 4.5
Figure 4.27 The complete forward pass of the DeepSeekMoE module, showing the three parallel data paths: the dense shared expert path, the sparse routed expert path with dynamic load balancing, and the residual connection.

Trace the shared expert branch, sparse routed branch, and residual connection as parallel paths whose outputs are summed. section 4.5

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Implements logits, biasing, top-k selection, gating, indexed dispatch, and bias updates, without detailing distributed execution.
Examples
Uses DeepSeekMoE forward-pass diagrams and PyTorch listings for expert construction, routing, dispatch, and bias updates.
What you can do after
Build it from scratch as a PyTorch sparse routing branch with configurable experts, top-k selection, optional mixed-precision routing, and load balancing.
How it connects
Connects mixture-of-experts routing with expert feed-forward networks, top-k selection, load balancing, shared experts, and residual connections.

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.