chapter thirteen

13 Capstone — cascading specialists into a model family for agents

 

This chapter covers

  • Curating a function-calling dataset
  • Specializing a model with LoRA to create the teacher model
  • Combining depth and width pruning with knowledge distillation
  • Iterate to derive a full model family, completing the cascade
  • Accelerating the specialist with speculative decoding
  • Routing requests across the family with structural checks and token confidence

LLM-based agents are part of my day to day, since I use them for programming support and as personal assistants. Even this book, which has stayed on the sidelines and focused on the structure of Transformer models, is going to give them a presence through this chapter.

An agent works as a chain of calls in which the model decides, step by step, which tool to invoke and with which arguments. Frequently, all those requests travel to the API of a frontier model, accumulating cost and latency with each iteration. In my projects building systems of this type, I've observed that many of the routing decisions are in fact narrow, repetitive tasks that could be resolved with much simpler models.

13.1 Designing the model family

13.2 Presenting the dataset

13.2.1 Preparing the function calling data

13.2.2 Preparing the examples for the model

13.3 Creating the specialist with LoRA

13.4 Creating the medium model through pruning and distillation

13.4.1 Pruning the specialist

13.4.2 Recovering knowledge

13.5 Creating the smaller model

13.6 Putting the family to work

13.6.1 Accelerating the specialist with speculative decoding

13.6.2 Cascaded routing with structural verification

13.7 Hands-on lab

13.8 Summary