liveBook · Manning

Chapter 13: Capstone — cascading specialists into a model family for agents

What you will learn in chapter 13 of Rearchitecting LLMs

This introduction motivates using simpler models for repetitive tool-routing decisions in LLM-based agents.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction motivates using simpler models for repetitive tool-routing decisions in LLM-based agents.

Key terms: agent, tool, arguments, frontier model, routing decisions

13.1 Designing the model family

This section motivates a three-model specialist family for agent calls and explains its cascading derivation and the limits of the evidence for that design.

Key terms: agent, external tools, frontier model, Teacher, Medium, Small, pruning, knowledge distillation, fine-tuning, cascade

13.2 Presenting the dataset

The section introduces xLAM as a function-calling dataset and narrows its broad coverage to three functions for a weather specialist.

Key terms: xLAM, function calls, user query, available tools, expected call, postal code, IP, city, weather

13.2.1 Preparing the function calling data

This section curates xLAM function-calling records, removes duplicate queries, and creates a function-stratified training and test split.

Key terms: xLAM, tools, answers, distractors, ALLOWED_FUNCTIONS, single call, normalize_query, deduplicate_by_query, data leakage, train_test_split

13.2.2 Preparing the examples for the model

This section prepares xLAM examples in Qwen3’s chat format, separating prompts from tool-call completions and preserving a common test set.

Key terms: chat template, tokenizer.chat_template, xLAM, tools, build_prompt_completion, full_text, prompt_text, prompt, completion, <tool_call>

13.3 Creating the specialist with LoRA

This section trains a LoRA-based function-calling specialist from Qwen3-0.6B and evaluates it against the original model.

Key terms: LoRA, Qwen3-0.6B, model T, Teacher, adapter, function calling, float16, quantization, low_cpu_mem_usage, use_cache

13.4 Creating the medium model through pruning and distillation

This section introduces the planned creation of model M from specialist T through pruning and knowledge distillation.

13.4.1 Pruning the specialist

This section prunes a fine-tuned specialist by depth and width using calibration data, then evaluates the resulting model’s function-calling degradation.

Key terms: calibration dataset, prompts, dynamic padding, attention_mask, cosine distance, passive blocks, protection heuristics, depth pruning, teacher, PPM

13.4.2 Recovering knowledge

The section prepares completion-only distillation data, recovers the pruned medium model using teacher guidance, and evaluates recovery across multiple seeds.

Key terms: input_ids, labels, -100, build_distillation_example, collate_distillation, attention_mask, pad_token_id, DataLoader, shuffle, task loss

13.5 Creating the smaller model

This section creates the smallest model S from M and examines its distillation, accuracy, and speed trade-offs.

Key terms: S, M, T, depth pruning, width pruning, knowledge distillation, gamma, beta, intermediate states, logits

13.6 Putting the family to work

This section introduces two ways a related family of models can cooperate to address agent latency, while cautioning that the demonstration does not predict production gains.

Key terms: deployment footprint, latency bottleneck, speculative decoding, cascaded routing, tokenizer, vocabulary, base lineage, production, T4 GPU, test examples

13.6.1 Accelerating the specialist with speculative decoding

This section explains speculative decoding with the model family and measures its speed and output fidelity against the target model alone.

Key terms: speculative decoding, draft model, target model, greedy decoding, acceptance rate, α, c, draft step, structural tokens, timed_generate

13.6.2 Cascaded routing with structural verification

This section implements a small-model-first cascade that routes structured tool calls using syntax and confidence checks, then evaluates its quality and latency trade-offs.

Key terms: cascaded routing, parse_tool_call, CONF_THRESHOLD, target_model, output_logits, min_prob, weakest_token, mean latency, escalation, speculative decoding

13.7 Hands-on lab

13.8 Summary

Concepts explained in this chapter

← Chapter 12: Case study — replacing one agent call with a specialized SLM in production

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.