A custom model is adapted to a narrow domain, dataset, or task rather than retaining broad general-purpose capability. A smaller specialized language model can replace an oversized model when the workload needs only a limited part of its capabilities, potentially reducing cost and inference latency. section 1.1.1 section 12.1
In an agent loop, a frequent call that extracts structured tool arguments is a candidate for specialization. The existing frontier model can serve both as a teacher and as the accuracy baseline for its replacement. The rest of the agent loop need not change. section 12.0 section 12.1
A practical specialization workflow has four stages:
- Capture exact prompts and responses from the target call. Reject malformed responses and flag examples whose downstream execution confirms the call. section 12.2
- Measure the frontier API and base model on the task. Use the specialization set to guide depth pruning, width pruning, and selective attention removal. section 12.3
- Recover accuracy by training the pruned model on the teacher’s completed responses. Check provider terms before using API outputs for training. section 12.3
- Before routing users to the replacement, shadow-test it on live traffic while users continue to receive the API’s answers. section 12.0
{"input": "Book a table for two tomorrow at 8 pm",
"target": {"tool_name": "book_table",
"parameters": {"party_size": 2, "time": "20:00"}},
"confirmed": true}A specialization record pairs the agent’s input with the teacher’s structured output and a downstream confirmation flag. section 12.2
Pruning can temporarily reduce task accuracy, so recovery must be measured against both the starting model and the teacher. In one structured argument-extraction case, width was aligned to a 128-unit hardware tile to avoid padding waste; attention removal also reduced the KV cache. The recovered model’s held-out accuracy was 0.962, versus 0.924 for the base model and 0.971 for the API. section 12.3
Table 12.1 Base vs. specialized model architecture
|
Property
|
Base Model
(Llama-3.2-3B)
|
Specialized
SLM
|
|
Parameters
|
3.21B
|
1,83B
|
|
Transformer
blocks
|
28
|
22 (after
depth pruning)
|
|
Feed-forward
intermediate size
|
8,192
|
5,120
(aligned to tile multiple 128)
|
|
Blocks with
attention
|
28
|
16 (6
removed with their blocks, 6 more stripped)
|
|
Key-value
heads
|
8
|
8
|
|
Task
accuracy (held-out)
|
0.924
|
0.962 (API
bar:0.971)
|
|
Size on disk
(FP16)
|
6.42
|
3,67 GB
|
Compare the base and specialized model’s topology, size, and held-out accuracy. section 12.3
Specialists can also form a model family: fine-tune a specialist, derive smaller models through pruning and knowledge distillation, and use the family with speculative decoding or difficulty-based routing. section 13.0



