liveBook · Manning

Shared expert

As explained in Build a DeepSeek Model (From Scratch)

A shared expert is an expert subnetwork in a mixture-of-experts architecture that is available across routed inputs rather than selected as a mutually exclusive expert.

A shared expert is a feed-forward subnetwork in a mixture-of-experts (MoE) layer that processes every input token, independently of the router. It provides a dense path alongside the sparse path of selected routed experts. section 4.4.3

The motivation is to reduce duplicated knowledge. For example, experts specializing in code and legal contracts may both need English grammar. Shared experts are intended to carry common knowledge, leaving routed experts more capacity for specialization. These domain labels are intuition, not preset expert assignments: specialization emerges during training. section 4.4.3 section 4.1.3

Figure 4.21 The DeepSeekMoE architecture with Shared and Routed Experts. All tokens are processed by the dense Shared Experts, while the router selectively sends each token to a sparse subset of the Routed Experts.

Follow each token through the dense shared path and the router-selected sparse path. section 4.4.3

For token x, combine the paths as: output = residual(x) + sum(shared expert outputs) + weighted sum(selected routed expert outputs). The router selects the top-k routed experts; it does not select whether shared experts run. section 4.4.3

A practical shared expert can be a two-layer feed-forward network: project to a hidden dimension, apply GELU and dropout, then project back to the model dimension. In the forward pass, initialize shared_out with zeros shaped like x and add each shared expert’s output. For an input shaped [batch, sequence, model dimension], this preserves the input shape for the final sum. section 4.5

Listing 4.3 The Shared Expert Path in the forward Method
   def forward(self, x: torch.Tensor) -> torch.Tensor:
        B, S, D = x.shape
        x_flat = x.reshape(-1, D)  # [N, D] with N=B*S
 
        # 1) shared path
        shared_out = torch.zeros_like(x)
        for exp in self.shared:
            shared_out += exp(x)
 
        # ... (routed path comes next)

The shared-path loop applies every shared expert to the full input and accumulates its output. section 4.5

Matched-parameter comparison

On TinyStories, a DeepSeek-MoE model with shared experts and dynamic balancing achieved best validation loss 1.9451, versus 1.9854 for a standard MoE, with about 101.3 million parameters each. The comparison changes more than shared experts, so it does not isolate their contribution. section 4.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains the three-path output and implements shared expert accumulation with a two-layer feed-forward network.
Examples
Uses grammar shared by code and legal-contract specialists, plus a TinyStories comparison of two MoE models.
What you can do after
Build it from scratch as a PyTorch shared path and combine it with routed and residual outputs.
How it connects
Connects mixture-of-experts routing, feed-forward networks, residual connections, and expert specialization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.