Load violation measures how far an expert’s routing load falls below or above the average load. It supplies a feedback signal for balancing mixture-of-experts routing without adding a balancing loss: underused experts receive a higher routing bias, while overused experts receive a lower one. section 4.4.4
For expert i, calculate violationᵢ = average load − actual loadᵢ. Count each token–expert selection as one assignment, then divide the total assignments by the number of experts. A positive violation means underloaded; a negative violation means overloaded; zero means the expert matches the average. section 4.4.4
Three experts
If experts receive 2, 2, and 4 token assignments, their average load is 8/3 ≈ 2.67. Their violations are approximately +0.67, +0.67, and −1.33. The first two biases should increase, and the third should decrease. section 4.4.4
A simple update adds u × sign(violationᵢ) to each persistent expert bias, where u is the update size. The bias is added to router logits before top-k selection, influencing the next routing decision. Biases start at zero and are updated after each training step. section 4.4.4
Listing 4.5 The update_bias Method for Load Balancing
@torch.no_grad()
def update_bias(self, x: torch.Tensor):
# Call once per optimizer step on the same tokens seen by forward.
# Uses the SAME logits (including current bias) to estimate loads.
N = x.shape[0] * x.shape[1]
logits = F.linear(x.reshape(-1,
self.d_model), self.centroids) + self.bias #A
_, idx = torch.topk(logits, self.top_k, dim=-1)
counts = torch.bincount(idx.flatten(),
minlength=self.n_routed).float() #B
avg = counts.sum() / max(1, self.n_routed)
# Smooth, bounded update; avoids large jumps from sign()
# violation > 0 => under-loaded (we want to increase its prior)
violation = (avg - counts) / (avg + 1e-6) #C
self.bias.add_(self.bias_lr * torch.tanh(violation)) #DThis PyTorch implementation recomputes biased top-k selections, counts assignments with torch.bincount, and updates the bias once per optimizer step without gradients. It normalizes the violation by the average load and applies torch.tanh for a smooth, bounded adjustment rather than using sign. section 4.5
