Expert load is the number or fraction of input tokens dispatched to each expert in a mixture-of-experts (MoE) computation. Uneven loads can create memory problems and leave expert networks inefficiently used. Load measures actual routing decisions, not just the router's probabilities. section 4.3.2 section 4.4.4
Equal importance, unequal work
Expert 1 receives one token with probability 1.0; expert 2 receives four tokens with probability 0.25 each. Both have importance 1.0, the sum of their probabilities, but their loads are one and four tokens. Balancing importance alone can therefore miss a fourfold workload difference. section 4.3.2
A load-balancing loss combines actual dispatch and router probability: λN Σᵢ fᵢpᵢ, where N is the number of experts, fᵢ is the fraction of batch tokens dispatched to expert i, pᵢ is its mean router probability, and λ controls the loss's contribution. A small λ can leave imbalance unchecked; a large one can divert training from the main objective. section 4.3.2 section 4.4.4
Auxiliary-loss-free balancing instead uses load as feedback for routing. After each training step, compare each expert's count with the average: increase the routing bias for an underloaded expert and decrease it for an overloaded one. The adjusted bias affects selection on the next step without being learned through backpropagation. section 4.4.4
Listing 4.5 The update_bias Method for Load Balancing
@torch.no_grad()
def update_bias(self, x: torch.Tensor):
# Call once per optimizer step on the same tokens seen by forward.
# Uses the SAME logits (including current bias) to estimate loads.
N = x.shape[0] * x.shape[1]
logits = F.linear(x.reshape(-1,
self.d_model), self.centroids) + self.bias #A
_, idx = torch.topk(logits, self.top_k, dim=-1)
counts = torch.bincount(idx.flatten(),
minlength=self.n_routed).float() #B
avg = counts.sum() / max(1, self.n_routed)
# Smooth, bounded update; avoids large jumps from sign()
# violation > 0 => under-loaded (we want to increase its prior)
violation = (avg - counts) / (avg + 1e-6) #C
self.bias.add_(self.bias_lr * torch.tanh(violation)) #DThe PyTorch update recomputes biased router logits, selects top-k experts, and uses torch.bincount to count assignments. It compares counts with their average and applies a tanh-smoothed bias update under torch.no_grad(). section 4.5
