liveBook · Manning

Soft routing

As explained in Rearchitecting LLMs

Soft routing is a mixture-of-experts routing strategy that combines the outputs of multiple experts using continuous routing weights.

Soft routing combines the outputs of multiple mixture-of-experts (MoE) experts using continuous weights for each token. In a two-expert model, both experts run, so this approach does not reduce computation by selectively activating experts. section 9.0 section 9.2.2

The routing weights can vary with token context:

  • On a Barcelona question, tokens mildly favor the general expert, with weights around 0.55–0.60. section 9.2.2
  • In a clinical note, weights shift toward the clinical expert as relevant context appears; “with” reaches 0.62. section 9.2.2
  • Hard routing instead processes each token with one expert, providing selective activation. section 9.2.2
Listing 9.7 Analyze routing
def analyze_routing(text, layer_index=0):
    inputs = tokenizer(text, return_tensors="pt").to(DEVICE)
    with torch.no_grad():
        with torch.amp.autocast("cuda", dtype=torch.float16):
            _ = model(**inputs)
 
    moe_layer      = model.model.layers[layer_index].mlp #A
    routing_weights = moe_layer.last_routing.squeeze(0) #B
    tokens          = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
 
    print(f"Routing analysis (layer {layer_index}):")
    print(f"{'Token':<18} | {'Expert 0 (General)':<20} | {'Expert 1 (Clinical)'}")
    print("-" * 65)
    for token, weights in zip(tokens, routing_weights):
        clean = token.replace("Ġ", "").replace(" ", "")
        print(f"{clean:<18} | {weights[0].item():<20.4f} | {weights[1].item():.4f}")

Inspect per-token routing weights with this analysis function. section 9.2.2

How Rearchitecting LLMs covers it

How deep it goes
Explains weighted expert combination and illustrates token-level routing preferences with observed weights.
Examples
Contrasts routing weights for a Barcelona question and a clinical note, then compares generated responses.
What you can do after
Diagnose token-level expert preferences by inspecting routing weights with analyze_routing.
How it connects
Connects mixture of experts to expert specialization, router weights, hard routing, and selective activation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.