liveBook · Manning

Shared path

As explained in Build a DeepSeek Model (From Scratch)

A shared path is the dense MoE computation in which every token is processed by a common set of experts.

A shared path is the dense branch of a mixture-of-experts layer: every token passes through the same small set of generalist experts, applying common foundational knowledge consistently. section 4.5

In the DeepSeekMoE forward pass, the shared output runs alongside the sparse routed path and residual connection; the three outputs are summed as x + shared_out + routed_out. section 4.5

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Implements expert initialization and tensor accumulation, while deferring routing and load balancing to separate paths.
Examples
Uses DeepSeekMoE diagrams, ExpertFFN modules, and PyTorch forward code that sums outputs from shared experts.
What it assumes
Assumes familiarity with tensors and feed-forward networks while grounding the branch through incremental PyTorch modules.
What you can do after
Build it from scratch as a PyTorch shared-expert branch within a complete DeepSeekMoE module.
How it connects
Connects mixture-of-experts layers with feed-forward networks, sparse routing, residual connections, and load balancing.

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.