liveBook · Manning

Custom model

As explained in Rearchitecting LLMs

A specialized model is a machine-learning model adapted for a particular domain, dataset, or set of tasks and assigned as an expert within a larger model.

A custom model is adapted to a narrow domain, dataset, or task rather than retaining broad general-purpose capability. A smaller specialized language model can replace an oversized model when the workload needs only a limited part of its capabilities, potentially reducing cost and inference latency. section 1.1.1 section 12.1

In an agent loop, a frequent call that extracts structured tool arguments is a candidate for specialization. The existing frontier model can serve both as a teacher and as the accuracy baseline for its replacement. The rest of the agent loop need not change. section 12.0 section 12.1

A practical specialization workflow has four stages:

  • Capture exact prompts and responses from the target call. Reject malformed responses and flag examples whose downstream execution confirms the call. section 12.2
  • Measure the frontier API and base model on the task. Use the specialization set to guide depth pruning, width pruning, and selective attention removal. section 12.3
  • Recover accuracy by training the pruned model on the teacher’s completed responses. Check provider terms before using API outputs for training. section 12.3
  • Before routing users to the replacement, shadow-test it on live traffic while users continue to receive the API’s answers. section 12.0
{"input": "Book a table for two tomorrow at 8 pm",
 "target": {"tool_name": "book_table",
            "parameters": {"party_size": 2, "time": "20:00"}},
 "confirmed": true}

A specialization record pairs the agent’s input with the teacher’s structured output and a downstream confirmation flag. section 12.2

Pruning can temporarily reduce task accuracy, so recovery must be measured against both the starting model and the teacher. In one structured argument-extraction case, width was aligned to a 128-unit hardware tile to avoid padding waste; attention removal also reduced the KV cache. The recovered model’s held-out accuracy was 0.962, versus 0.924 for the base model and 0.971 for the API. section 12.3

Table 12.1 Base vs. specialized model architecture
Property
Base Model
(Llama-3.2-3B)
Specialized SLM
Parameters
3.21B
1,83B
Transformer blocks
28
22 (after depth pruning)
Feed-forward intermediate size
8,192
5,120 (aligned to tile multiple 128)
Blocks with attention
28
16 (6 removed with their blocks, 6 more stripped)
Key-value heads
8
8
Task accuracy (held-out)
0.924
0.962 (API bar:0.971)
Size on disk (FP16)
6.42
3,67 GB

Compare the base and specialized model’s topology, size, and held-out accuracy. section 12.3

Specialists can also form a model family: fine-tune a specialist, derive smaller models through pruning and knowledge distillation, and use the family with speculative decoding or difficulty-based routing. section 13.0

How Rearchitecting LLMs covers it

How deep it goes
Details traffic filtering, task-specific pruning, recovery training, and a measured reference topology.
Examples
Uses structured tool-argument extraction and a confirmed table-booking request to illustrate specialization.
What you can do after
Evaluate a specialized model against a frontier API using held-out accuracy and shadow testing.
How it connects
Connects structured pruning and knowledge distillation to speculative decoding and difficulty-based routing.

Also explained in

Compare how 4 books explain custom model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.