liveBook · Manning

Load violation

As explained in Build a DeepSeek Model (From Scratch)

Load violation is a signed measure of an expert's deviation from the desired routing load, distinguishing under-utilization from over-utilization.

Load violation measures how far an expert’s routing load falls below or above the average load. It supplies a feedback signal for balancing mixture-of-experts routing without adding a balancing loss: underused experts receive a higher routing bias, while overused experts receive a lower one. section 4.4.4

For expert i, calculate violationᵢ = average load − actual loadᵢ. Count each token–expert selection as one assignment, then divide the total assignments by the number of experts. A positive violation means underloaded; a negative violation means overloaded; zero means the expert matches the average. section 4.4.4

Three experts

If experts receive 2, 2, and 4 token assignments, their average load is 8/3 ≈ 2.67. Their violations are approximately +0.67, +0.67, and −1.33. The first two biases should increase, and the third should decrease. section 4.4.4

A simple update adds u × sign(violationᵢ) to each persistent expert bias, where u is the update size. The bias is added to router logits before top-k selection, influencing the next routing decision. Biases start at zero and are updated after each training step. section 4.4.4

Listing 4.5 The update_bias Method for Load Balancing
@torch.no_grad()
def update_bias(self, x: torch.Tensor):
    # Call once per optimizer step on the same tokens seen by forward.
    # Uses the SAME logits (including current bias) to estimate loads.
    
    N = x.shape[0] * x.shape[1]
    logits = F.linear(x.reshape(-1,
        self.d_model), self.centroids) + self.bias  #A
    
    _, idx = torch.topk(logits, self.top_k, dim=-1)
 
    counts = torch.bincount(idx.flatten(),
        minlength=self.n_routed).float()  #B
    
    avg = counts.sum() / max(1, self.n_routed)
 
    # Smooth, bounded update; avoids large jumps from sign()
    # violation > 0 => under-loaded (we want to increase its prior)
    violation = (avg - counts) / (avg + 1e-6)  #C
 
    self.bias.add_(self.bias_lr * torch.tanh(violation))  #D

This PyTorch implementation recomputes biased top-k selections, counts assignments with torch.bincount, and updates the bias once per optimizer step without gradients. It normalizes the violation by the average load and applies torch.tanh for a smooth, bounded adjustment rather than using sign. section 4.5

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Derives the signed deviation and bias update, then implements a normalized, smoothed update in PyTorch.
Examples
Calculates violations for experts with loads of 2, 2, and 4, and implements bias updates in a DeepSeekMoE layer.
What you can do after
Build it from scratch in PyTorch to update expert routing biases from observed token assignments.
How it connects
Connects mixture-of-experts routing, top-k selection, router logits, and dynamic bias adjustment.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.