liveBook · Manning

Routing

As explained in Build a DeepSeek Model (From Scratch)

A router is a small learnable neural network that assigns scores to experts for each input token in a mixture-of-experts model.

Routing lets a mixture-of-experts (MoE) layer use only a subset of its expert networks for each token. A small learnable router scores every expert; top-k selection chooses which experts run, avoiding computation for the rest. section 4.2.3 section 4.2.2

The router can be a linear layer. For input X with shape (tokens, embedding dimension) and learned routing matrix W with shape (embedding dimension, experts), the logits XW have shape (tokens, experts). Each entry is a raw suitability score for one token–expert pair. section 4.2.3

Four tokens, three experts

Four 8-dimensional token embeddings form a (4, 8) input matrix. Multiplying by an (8, 3) routing matrix produces a (4, 3) score matrix. With k=2, each row selects two experts; the third expert does not process that token. section 4.2.3 section 4.2.2

Figure 4.7 The routing mechanism. The input matrix is multiplied by a learned routing matrix to produce an expert selector matrix, which contains a raw score for each expert for each token.

Follow the matrix multiplication from token embeddings to per-expert logits. section 4.2.3

A routed forward pass can be implemented in four steps:

  • Flatten a batch of token embeddings and calculate router logits with a learned linear projection. section 4.5
  • Add the current per-expert balancing bias, select the top k logits per token, and apply softmax to those selected logits to obtain gate weights. section 6.2.1
  • For each expert, gather the tokens that selected it, run the expert, and accumulate its weighted outputs at the corresponding token positions. section 4.5 section 6.2.1
  • Add the routed result to the shared-expert result. Shared experts process every token; routed experts process only selected tokens. section 4.5 section 6.2.1
Listing 6.12 The forward method of DeepSeekMoE (Routing and Dispatch)
def forward(self, x: torch.Tensor):
        B, S, D = x.shape
        x_flat = x.reshape(-1, D)        
        # Shared expert path (dense)
        shared_out = torch.zeros_like(x)
        for exp in self.shared_experts: shared_out += exp(x)        
        # --- Routed expert path (sparse) —
        router_logits = self.gate(x_flat) #A
        router_logits_with_bias = router_logits + self.bias.to(router_logits.dtype) #B
        top_k_logits, top_k_indices = torch.topk(router_logits_with_bias, self.top_k, dim=-1)
        gates = F.softmax(top_k_logits, dim=-1, dtype=torch.float).type_as(x) #C     
        routed_out_flat = torch.zeros_like(x_flat)
        for i in range(self.n_routed): #C
            mask = (top_k_indices == i)
            row_idx, which_k = torch.where(mask)
            if row_idx.numel() == 0:
                continue
            w = gates[row_idx, which_k].unsqueeze(-1)
            exp_in = x_flat[row_idx]
            exp_out = self.routed_experts[i](exp_in)
            routed_out_flat.index_add_(0, row_idx, exp_out * w)        # ... (bias update and final output comes next)

Inspect the PyTorch routing and dispatch path, including tensor reshaping, top-k selection, and gate weights. section 6.2.1

Sparse routing can become imbalanced. Experts selected early receive more training and may become increasingly favored, while rarely selected experts get few or no updates. Overused experts can also become bottlenecks. section 4.3

One balancing approach adds an auxiliary loss, but its weight trades balance against the main training objective. Auxiliary-loss-free balancing instead adjusts a non-trainable per-expert bias: increase it for underloaded experts and decrease it for overloaded ones, influencing the next top-k selection. section 4.4.4

Listing 4.5 The update_bias Method for Load Balancing
@torch.no_grad()
def update_bias(self, x: torch.Tensor):
    # Call once per optimizer step on the same tokens seen by forward.
    # Uses the SAME logits (including current bias) to estimate loads.
    
    N = x.shape[0] * x.shape[1]
    logits = F.linear(x.reshape(-1,
        self.d_model), self.centroids) + self.bias  #A
    
    _, idx = torch.topk(logits, self.top_k, dim=-1)
 
    counts = torch.bincount(idx.flatten(),
        minlength=self.n_routed).float()  #B
    
    avg = counts.sum() / max(1, self.n_routed)
 
    # Smooth, bounded update; avoids large jumps from sign()
    # violation > 0 => under-loaded (we want to increase its prior)
    violation = (avg - counts) / (avg + 1e-6)  #C
 
    self.bias.add_(self.bias_lr * torch.tanh(violation))  #D

See a no-gradient bias update that counts selections, compares them with average load, and applies a bounded adjustment. section 4.5

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Develops routing from matrix dimensions through top-k dispatch and dynamic load-balancing updates in a complete MoE implementation.
Examples
Uses four 8-dimensional token embeddings and three experts to illustrate router score dimensions and two-of-three selection.
What you can do after
Builds routing from scratch in PyTorch, including expert dispatch and dynamic bias updates.
How it connects
Connects mixture of experts with expert specialization, softmax gating, and load balancing.

Also explained in

Compare how 34 books explain routing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.