A small language model (SLM) typically has a few million to a few billion parameters. Its smaller size makes it a candidate for fast, lightweight deployment, especially when a task does not need the full breadth of a general-purpose model. section 1.0 section 12.1
Specialization matches the model to a task, its data, and deployment constraints. Instead of relying only on prompts, rearchitecting physically changes a pretrained model—for example, by reducing its parameters or layers. Structured pruning removes capacity; knowledge distillation retrains the resulting model to recover accuracy. section 1.0 section 6.6
A practical specialization workflow follows these stages:
- Capture data from the target task and measure the existing model’s performance. section 12.1
- Prune the model to a smaller shape, then retrain it against the desired performance bar. The model being replaced can provide both teacher outputs and a baseline. section 12.1 section 6.6
- Validate deployment under real traffic before routing users to the replacement. Check output validity, latency, cost, and reliability; retain an API fallback where appropriate. section 12.0
Distillation can combine task loss from hard dataset labels with a logits loss from the teacher’s soft labels. The balance is task-dependent: reported large-scale comparisons favor distillation over ordinary fine-tuning after pruning, while one comparison finds a pure logits objective stronger than hybrid objectives. section 6.6
Replacing an agent call
An agent repeatedly calls a frontier API to turn a short prompt into typed tool arguments. A self-hosted SLM specialized for that call can target the recurring per-token cost and blocking network latency without replacing the rest of the agent loop. Shadow testing sends live traffic to the candidate while users continue receiving the API’s responses. section 12.0 section 12.1
An SLM can also use a small mixture of experts to specialize in a few tasks. In one example, the resulting specialist improved on several general benchmarks but dropped slightly on others, so specialization still requires evaluation against the base model. section 9.3.3
Figure 9.5 Two-phase training strategy for the upcycled MoE. Phase 1 trains only the router across all 24 layers (lr = 1e-3) for three epochs — 98,304 parameters, 0.0034% of the model — reaching 90% schema compliance. Phase 2 reduces the router learning rate to lr = 1e-6 and unfreezes the expert_clinical tail layers 18–23 at lr = 1e-5 for one epoch, bringing trainable parameters to 10.35% and compliance to 100%.
The two-phase training diagram shows how router training and expert-tail fine-tuning are separated in a small mixture of experts. section 9.3.3


