liveBook · Manning

Expert load

As explained in Build a DeepSeek Model (From Scratch)

Expert load is the number or proportion of input tokens routed to each expert during an MoE computation.

Expert load is the number or fraction of input tokens dispatched to each expert in a mixture-of-experts (MoE) computation. Uneven loads can create memory problems and leave expert networks inefficiently used. Load measures actual routing decisions, not just the router's probabilities. section 4.3.2 section 4.4.4

Equal importance, unequal work

Expert 1 receives one token with probability 1.0; expert 2 receives four tokens with probability 0.25 each. Both have importance 1.0, the sum of their probabilities, but their loads are one and four tokens. Balancing importance alone can therefore miss a fourfold workload difference. section 4.3.2

A load-balancing loss combines actual dispatch and router probability: λN Σᵢ fᵢpᵢ, where N is the number of experts, fᵢ is the fraction of batch tokens dispatched to expert i, pᵢ is its mean router probability, and λ controls the loss's contribution. A small λ can leave imbalance unchecked; a large one can divert training from the main objective. section 4.3.2 section 4.4.4

Auxiliary-loss-free balancing instead uses load as feedback for routing. After each training step, compare each expert's count with the average: increase the routing bias for an underloaded expert and decrease it for an overloaded one. The adjusted bias affects selection on the next step without being learned through backpropagation. section 4.4.4

Listing 4.5 The update_bias Method for Load Balancing
@torch.no_grad()
def update_bias(self, x: torch.Tensor):
    # Call once per optimizer step on the same tokens seen by forward.
    # Uses the SAME logits (including current bias) to estimate loads.
    
    N = x.shape[0] * x.shape[1]
    logits = F.linear(x.reshape(-1,
        self.d_model), self.centroids) + self.bias  #A
    
    _, idx = torch.topk(logits, self.top_k, dim=-1)
 
    counts = torch.bincount(idx.flatten(),
        minlength=self.n_routed).float()  #B
    
    avg = counts.sum() / max(1, self.n_routed)
 
    # Smooth, bounded update; avoids large jumps from sign()
    # violation > 0 => under-loaded (we want to increase its prior)
    violation = (avg - counts) / (avg + 1e-6)  #C
 
    self.bias.add_(self.bias_lr * torch.tanh(violation))  #D

The PyTorch update recomputes biased router logits, selects top-k experts, and uses torch.bincount to count assignments. It compares counts with their average and applies a tanh-smoothed bias update under torch.no_grad(). section 4.5

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines load against importance, derives a balancing-loss expression, and implements a per-step bias update.
Examples
Contrasts equal importance with one versus four dispatched tokens and counts top-k assignments in a PyTorch MoE module.
What you can do after
Builds a load-balancing bias update from scratch using PyTorch token counts and smoothed feedback.
How it connects
Connects expert load to router probabilities, top-k routing, auxiliary losses, and dynamic bias adjustment.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.