liveBook · Manning

Expert network

As explained in Rearchitecting LLMs

An expert is an independently parameterized neural-network component in a mixture-of-experts architecture that processes selected inputs.

An expert is an independently parameterized network component in a mixture-of-experts (MoE) model. Experts process inputs separately; a router assigns weights based on each token’s hidden state, and the forward pass combines their outputs. Experts may specialize, but specialization need not follow domain boundaries. section 9.2 section 9.4.2

In a Transformer MoE, an expert can be an MLP replacing part of the original feed-forward path. Soft routing computes a weighted mixture of experts. Top-k routing instead selects a fixed number per token and renormalizes their weights, allowing unselected experts to be skipped for that token. The router uses contextual hidden states, so the same token can be routed differently in different contexts. section 9.3.1 section 9.6

Figure 9.4 Two views of the same top-k routing decision. Left: the router produces one logit score per expert for each token; the two highest scores per row (highlighted) are selected. Right: the same assignments reorganized by expert, showing which tokens each expert will process and their renormalized weights — for each token, those weights sum to 1 across its assigned experts. The example uses four experts rather than two so that top_k = 2 leaves half the experts idle per token. The loop in TrainableTopKMoE.forward performs exactly this regrouping.

Follow how token-level top-k selections are regrouped by expert and combined with renormalized weights. section 9.3.1

A practical two-expert construction uses sparse upcycling:

  • Wrap selected original MLPs in MoE blocks. Keep each original MLP as the general expert and deep-copy it to initialize a separately trainable clinical expert. section 9.2.1
  • Add a linear router from hidden-state size to two scores. Freeze the rest of the model, then train the clinical expert and router on clinical and general prompts. section 9.2.1
  • Inspect per-token routing weights and test both general responses and clinical schema compliance. A routing preference alone does not establish task quality. section 9.2.2
Listing 9.1 Trainable2ExpertMoE
class Trainable2ExpertMoE(nn.Module):
    def __init__(self, original_mlp, hidden_size):
        super().__init__()
        target_dtype = next(original_mlp.parameters()).dtype
        target_device = next(original_mlp.parameters()).device
 
        self.expert_general = original_mlp #A
        self.expert_clinical = copy.deepcopy(original_mlp) #B
        self.router = nn.Linear(hidden_size, 2, 
                                bias=False,
                                dtype=target_dtype, 
                                device=target_device) #C
        self.last_routing = None 
 
    def forward(self, hidden_states):
        router_logits = self.router(hidden_states)
 
        if self.training:
            noise = torch.randn_like(router_logits) * 0.35  #1
            router_logits = router_logits + noise
 
        routing_weights = F.softmax(
            router_logits.float(), dim=-1
        ).to(hidden_states.dtype)  #2
 
        out_general = self.expert_general(hidden_states)
        out_clinical = self.expert_clinical(hidden_states)
 
        # Soft routing: each token receives a weighted blend of both experts
        output = out_general * routing_weights[..., 0].unsqueeze(-1) 
[CA]+ out_clinical * routing_weights[..., 1].unsqueeze(-1) #3
        self.last_routing = routing_weights.detach() #D
 
        return output

Inspect the two-expert module, including the deep copy, router initialization, and forward-pass setup. section 9.2.1

Clinical extraction expert

In SmolLM2-1.7B-Instruct, MLPs in layers 13–23 are upcycled into frozen general and trainable clinical experts; attention remains unchanged. On a 40-sample clinical test set, the soft-routed model reaches 97.5% schema compliance. On irrelevant notes, compliance rises from 58.3% for the base model to 91.7%; a general knowledge prompt also receives a coherent answer. section 9.2 section 9.2.2

An alternative is to transplant MLPs from an already fine-tuned model and initially train only the router. This is cheaper to train, but the transplanted MLPs may have adapted to different attention layers and need further adjustment. In one router-only test, two-expert soft routing achieved 90.0% clinical schema compliance; the irrelevant-note category fell to 66.7%. section 9.3 section 9.3.2

More experts do not guarantee clean domain separation. Mixtral 8x7B selects two of eight experts per token; its observed routing showed syntactic patterns rather than clear domain specialization. A three-expert exercise tests general, clinical, and code experts by varying training depth and freezing, then comparing routing across layers and prompts. section 9.4.2 section 9.5

How Rearchitecting LLMs covers it

How deep it goes
Implements expert initialization, freezing, routing, training, transplantation, and evaluation, including top-k inference and compatibility checks.
Examples
Uses general and clinical MLP experts, a transplanted clinical expert, and a three-expert general, clinical, and Python-code exercise.
What it assumes
Assumes an existing base model and clinical evaluation functions while explaining expert construction and routing in detail.
What you can do after
Builds expert-based MoE blocks from existing MLPs, trains selected experts and routers, and inspects routing alongside task performance.
How it connects
Connects expert networks to mixture of experts, dynamic inference, sparse upcycling, top-k routing, and Transformer MLPs.

Also explained in

Compare how 5 books explain expert network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.