Soft routing combines the outputs of multiple mixture-of-experts (MoE) experts using continuous weights for each token. In a two-expert model, both experts run, so this approach does not reduce computation by selectively activating experts. section 9.0 section 9.2.2
The routing weights can vary with token context:
- On a Barcelona question, tokens mildly favor the general expert, with weights around 0.55–0.60. section 9.2.2
- In a clinical note, weights shift toward the clinical expert as relevant context appears; “with” reaches 0.62. section 9.2.2
- Hard routing instead processes each token with one expert, providing selective activation. section 9.2.2
Listing 9.7 Analyze routing
def analyze_routing(text, layer_index=0):
inputs = tokenizer(text, return_tensors="pt").to(DEVICE)
with torch.no_grad():
with torch.amp.autocast("cuda", dtype=torch.float16):
_ = model(**inputs)
moe_layer = model.model.layers[layer_index].mlp #A
routing_weights = moe_layer.last_routing.squeeze(0) #B
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
print(f"Routing analysis (layer {layer_index}):")
print(f"{'Token':<18} | {'Expert 0 (General)':<20} | {'Expert 1 (Clinical)'}")
print("-" * 65)
for token, weights in zip(tokens, routing_weights):
clean = token.replace("Ġ", "").replace(" ", "")
print(f"{clean:<18} | {weights[0].item():<20.4f} | {weights[1].item():.4f}")
Inspect per-token routing weights with this analysis function. section 9.2.2
