chapter thirteen
13 Capstone — cascading specialists into a model family for agents
This chapter covers
- Curating a function-calling dataset
- Specializing a model with LoRA to create the teacher model
- Combining depth and width pruning with knowledge distillation
- Iterate to derive a full model family, completing the cascade
- Accelerating the specialist with speculative decoding
- Routing requests across the family with structural checks and token confidence
LLM-based agents are part of my day to day, since I use them for programming support and as personal assistants. Even this book, which has stayed on the sidelines and focused on the structure of Transformer models, is going to give them a presence through this chapter.
An agent works as a chain of calls in which the model decides, step by step, which tool to invoke and with which arguments. Frequently, all those requests travel to the API of a frontier model, accumulating cost and latency with each iteration. In my projects building systems of this type, I've observed that many of the routing decisions are in fact narrow, repetitive tasks that could be resolved with much simpler models.