Chapter 13: Capstone — cascading specialists into a model family for agents
This introduction motivates using simpler models for repetitive tool-routing decisions in LLM-based agents.
After this chapter you will be able to
- Agent tool routing costs. Explain how an agent makes tool decisions across a chain of calls and why simpler models may be suitable for some of those decisions.
- Agent call efficiency motivation. Explain why agent workloads motivate replacing some frontier-model calls with smaller specialists.
- Three model family lineage. Describe how the Teacher, Medium, and Small models are derived from one another.
- Cascade evidence and limits. Explain the evidence offered for a pruning cascade and why it does not establish that cascading wins for this family.
- Function Calling Dataset Fit. Explain why xLAM is suitable for training the model and describe the contents of a record.
- Single call domain filtering. Explain how the records are filtered to the desired functions and how the function counts are checked.
- Query normalization and deduplication. Explain how normalized queries are deduplicated before splitting and why this helps prevent leakage.
- Function stratified data split. Explain why and how the deduplicated examples are split using function labels for stratification.
- Chat template formatting. Describe how Qwen3’s chat template formats an xLAM example for the model.
- Prompt and completion split. Explain how build_prompt_completion separates the model input from the tool call it should generate, and why the split is used.
- Function Calling Specialist. Describe how the post-trained Qwen3-0.6B checkpoint becomes the T specialist for the three selected functions.
- Completion Only Training. Explain how the training loss uses the dataset's prompt and completion split to focus specialization on function calls.
- Specialist Evaluation Results. Explain how the original model and T specialist are compared and what the results show about their function-calling errors.
- Prompt based calibration data. Explain how the calibration data is prepared and why it uses training prompts rather than completions.
- Protected depth pruning selection. Trace how calibration scores and protection rules select blocks for removal, and explain why pruning uses a copy of T.
- Hybrid MLP width pruning. Describe how hybrid PPM selects neurons and how the expansion rate sets the pruned MLP width.
- Post pruning evaluation results. Describe the function-calling evaluation after pruning and explain what its results imply for recovery.
- Completion-only distillation labels. Explain how the existing prompt-completion pairs are tokenized and labeled so distillation loss applies only to the tool call.
- Recovery loss configuration. Explain which signals guide the pruned student and why the teacher's logits receive greater weight.
- Multi-seed recovery evaluation. Explain how the three distillation runs are kept comparable and why seed 42 is selected as the canonical M in advance.
- Recovery results and limitations. Summarize the measured improvement from distillation and the remaining difference between M and T.
- Smallest model construction. Describe how S is created from M and how the notebook allows the distillation teacher to be chosen.
- Intermediate state loss adjustment. Explain why the S distillation loss adds intermediate-state alignment and how its weights change.
- Pruning failures and recovery. Distinguish the failures observed when pruning T to M from those when pruning M to S, and describe the distillation result.
- Independent Deployment Limitation. Explain why deploying T, M, and S as independent services does not by itself solve the agent's latency problem.
- Complementary Model Cooperation. Distinguish speculative decoding from cascaded routing as ways for the related models to cooperate.
- Draft and target verification. Explain how S and T generate tokens together and why greedy speculative decoding preserves T's output.
- Acceptance and draft cost. Explain how acceptance rate and relative draft-step cost affect speedup, and why the formulaic agent call is well suited to drafting.
- Fair generation timing. Describe how the notebook compares speculative generation with T alone and checks that the outputs agree.
- Cascade routing criteria. Trace how the router decides whether to return model S's tool call or regenerate it with model M or T.
- Weakest token confidence. Explain how the router measures S's confidence and why it uses the least confident token rather than an average.
- Cascade latency trade-off. Explain the cascade's approximate latency and how its use of the larger model differs from speculative decoding.
What it builds on
- One call in the agent's loop (chapter 12) · The role of a model call within an agent's loop.
- Recovering the knowledge (chapter 6) · Knowledge recovery through distillation.
- Specializing with QLoRa and QDoRA (chapter 7) · Specialization through fine-tuning.
- The foundations of LoRA (chapter 7) · The foundations of LoRA.
- Quantization and the memory bottleneck (chapter 7) · Quantization and the memory bottleneck.
- Choosing the blocks to discard (chapter 4) · Data-driven block selection and protective pruning heuristics.
- Reconstructing the MLP with hybrid scores (chapter 5) · Hybrid scores combine neuron weights with data-driven information.
- The compound loss function (chapter 6) · Distillation combines task and teacher-guidance losses.
Section by section
Introduction
This introduction motivates using simpler models for repetitive tool-routing decisions in LLM-based agents.
- Agent tool routing costs: Explain how an agent makes tool decisions across a chain of calls and why simpler models may be suitable for some of those decisions.
Why it matters: This explains why an agent may not need a frontier model for every decision.
Key terms: agent, tool, arguments, frontier model, routing decisions
13.1 Designing the model family
This section motivates a three-model specialist family for agent calls and explains its cascading derivation and the limits of the evidence for that design.
- Agent call efficiency motivation: Explain why agent workloads motivate replacing some frontier-model calls with smaller specialists.
Why it matters: Understanding the workload explains why replacing individual calls can be worthwhile. - Three model family lineage: Describe how the Teacher, Medium, and Small models are derived from one another.
Why it matters: The lineage shows how one specialist becomes a family of progressively smaller models. - Cascade evidence and limits: Explain the evidence offered for a pruning cascade and why it does not establish that cascading wins for this family.
Why it matters: The comparison motivates the design without guaranteeing it will outperform direct pruning in a different setting.
Key terms: agent, external tools, frontier model, Teacher, Medium, Small, pruning, knowledge distillation, fine-tuning, cascade
13.2 Presenting the dataset
The section introduces xLAM as a function-calling dataset and narrows its broad coverage to three functions for a weather specialist.
- Function Calling Dataset Fit: Explain why xLAM is suitable for training the model and describe the contents of a record.
Why it matters: The record structure supplies the inputs and expected output for the function-calling task. - Weather Specialist Data Selection: Describe why the example selects a subset of xLAM and identify the three functions it keeps.
Why it matters: Narrowing the data focuses the example on the functions needed for the specialist.
Key terms: xLAM, function calls, user query, available tools, expected call, postal code, IP, city, weather
13.2.1 Preparing the function calling data
This section curates xLAM function-calling records, removes duplicate queries, and creates a function-stratified training and test split.
- Function calling record structure: Describe the parts of an xLAM record and the selection task they present to the model.
Why it matters: The examples test tool selection as well as argument filling. - Single call domain filtering: Explain how the records are filtered to the desired functions and how the function counts are checked.
Why it matters: Filtering defines the specialist's task while checking that each selected function has examples. - Query normalization and deduplication: Explain how normalized queries are deduplicated before splitting and why this helps prevent leakage.
Why it matters: Removing duplicate queries before splitting reduces overlap between training and evaluation. - Function stratified data split: Explain why and how the deduplicated examples are split using function labels for stratification.
Why it matters: Stratification helps the evaluation set reflect the mix of functions rather than accidentally undersampling a rare one.
Key terms: xLAM, tools, answers, distractors, ALLOWED_FUNCTIONS, single call, normalize_query, deduplicate_by_query, data leakage, train_test_split
13.2.2 Preparing the examples for the model
This section prepares xLAM examples in Qwen3’s chat format, separating prompts from tool-call completions and preserving a common test set.
- Chat template formatting: Describe how Qwen3’s chat template formats an xLAM example for the model.
Why it matters: The template puts each example into the dialogue format the model expects. - Prompt and completion split: Explain how build_prompt_completion separates the model input from the tool call it should generate, and why the split is used.
Why it matters: The split distinguishes what the model is given from the tool call it is trained to produce. - Common untouched test set: Explain how the test examples are prepared and used across the model family.
Why it matters: A common untouched test set gives the models the same basis for measurement.
Key terms: chat template, tokenizer.chat_template, xLAM, tools, build_prompt_completion, full_text, prompt_text, prompt, completion, <tool_call>
13.3 Creating the specialist with LoRA
This section trains a LoRA-based function-calling specialist from Qwen3-0.6B and evaluates it against the original model.
- Function Calling Specialist: Describe how the post-trained Qwen3-0.6B checkpoint becomes the T specialist for the three selected functions.
Why it matters: This establishes the specialist that the rest of the model family builds on. - Training Memory Setup: Explain the main loading and memory choices used to prepare the model for LoRA training.
Why it matters: The setup makes training feasible without quantizing a model that already fits in memory. - LoRA Adapter Configuration: Describe the LoRA adapter settings and the parts of the model they target.
Why it matters: These settings define where and how the specialist adapters are trained. - Completion Only Training: Explain how the training loss uses the dataset's prompt and completion split to focus specialization on function calls.
Why it matters: The loss directs training toward the output the specialist must produce. - Specialist Evaluation Results: Explain how the original model and T specialist are compared and what the results show about their function-calling errors.
Why it matters: The comparison shows whether specialization improves the actual tool calls rather than merely producing valid JSON.
Key terms: LoRA, Qwen3-0.6B, model T, Teacher, adapter, function calling, float16, quantization, low_cpu_mem_usage, use_cache
13.4 Creating the medium model through pruning and distillation
This section introduces the planned creation of model M from specialist T through pruning and knowledge distillation.
13.4.1 Pruning the specialist
This section prunes a fine-tuned specialist by depth and width using calibration data, then evaluates the resulting model’s function-calling degradation.
- Prompt based calibration data: Explain how the calibration data is prepared and why it uses training prompts rather than completions.
Why it matters: Calibration makes pruning decisions reflect the inputs the specialist must handle. - Protected depth pruning selection: Trace how calibration scores and protection rules select blocks for removal, and explain why pruning uses a copy of T.
Why it matters: The selection removes relatively passive blocks while protecting sensitive regions and retaining an intact teacher. - Hybrid MLP width pruning: Describe how hybrid PPM selects neurons and how the expansion rate sets the pruned MLP width.
Why it matters: The hybrid ranking and explicit width target produce a smaller model shaped by the specialist’s inputs. - Post pruning evaluation results: Describe the function-calling evaluation after pruning and explain what its results imply for recovery.
Why it matters: The evaluation shows why a smaller architecture alone is not yet an acceptable replacement for the specialist.
Key terms: calibration dataset, prompts, dynamic padding, attention_mask, cosine distance, passive blocks, protection heuristics, depth pruning, teacher, PPM
13.4.2 Recovering knowledge
The section prepares completion-only distillation data, recovers the pruned medium model using teacher guidance, and evaluates recovery across multiple seeds.
- Completion-only distillation labels: Explain how the existing prompt-completion pairs are tokenized and labeled so distillation loss applies only to the tool call.
Why it matters: This preserves the prompt as context without training the student to reproduce it. - Distillation batch padding: Describe how distillation examples are padded and shuffled into training batches.
Why it matters: Correct padding excludes artificial positions from representations and loss while shuffling reduces order-dependent training patterns. - Recovery loss configuration: Explain which signals guide the pruned student and why the teacher's logits receive greater weight.
Why it matters: The chosen objective prioritizes the teacher's richer guidance without matching intermediate block states. - Distillation target softening: Describe the roles of temperature and skew in the distillation configuration.
Why it matters: These settings make teacher guidance more informative and temper the initial gap created by pruning. - Multi-seed recovery evaluation: Explain how the three distillation runs are kept comparable and why seed 42 is selected as the canonical M in advance.
Why it matters: Comparable runs reveal variability without turning the test set into a tool for selecting a lucky seed. - Recovery results and limitations: Summarize the measured improvement from distillation and the remaining difference between M and T.
Why it matters: The results show substantial recovery after pruning without implying that M matches the teacher on every argument.
Key terms: input_ids, labels, -100, build_distillation_example, collate_distillation, attention_mask, pad_token_id, DataLoader, shuffle, task loss
13.5 Creating the smaller model
This section creates the smallest model S from M and examines its distillation, accuracy, and speed trade-offs.
- Smallest model construction: Describe how S is created from M and how the notebook allows the distillation teacher to be chosen.
Why it matters: This explains how the smallest member extends the existing model family. - Intermediate state loss adjustment: Explain why the S distillation loss adds intermediate-state alignment and how its weights change.
Why it matters: The loss adjustment addresses the greater relative structural change in the smaller model. - Pruning failures and recovery: Distinguish the failures observed when pruning T to M from those when pruning M to S, and describe the distillation result.
Why it matters: Accuracy loss can reflect different failure modes, so recovery should be evaluated beyond whether output is valid. - Accuracy and speed trade-off: Describe the reported speed gain of S over T alongside the accuracy cost of pruning.
Why it matters: The speed benefit gives context to the accuracy cost of creating a smaller model.
Key terms: S, M, T, depth pruning, width pruning, knowledge distillation, gamma, beta, intermediate states, logits
13.6 Putting the family to work
This section introduces two ways a related family of models can cooperate to address agent latency, while cautioning that the demonstration does not predict production gains.
- Independent Deployment Limitation: Explain why deploying T, M, and S as independent services does not by itself solve the agent's latency problem.
Why it matters: The family needs a way to work together, not just three separate deployments. - Complementary Model Cooperation: Distinguish speculative decoding from cascaded routing as ways for the related models to cooperate.
Why it matters: The two patterns use the family differently: one speeds generation, while the other divides calls by difficulty. - Demonstration Scale Limits: Explain why the section's demonstration should not be used to predict production gains.
Why it matters: A small sequential test demonstrates implementation without establishing production performance.
Key terms: deployment footprint, latency bottleneck, speculative decoding, cascaded routing, tokenizer, vocabulary, base lineage, production, T4 GPU, test examples
13.6.1 Accelerating the specialist with speculative decoding
This section explains speculative decoding with the model family and measures its speed and output fidelity against the target model alone.
- Draft and target verification: Explain how S and T generate tokens together and why greedy speculative decoding preserves T's output.
Why it matters: This explains how a smaller model can accelerate generation without replacing the target's greedy decisions. - Acceptance and draft cost: Explain how acceptance rate and relative draft-step cost affect speedup, and why the formulaic agent call is well suited to drafting.
Why it matters: A highly accurate draft is not enough to guarantee a worthwhile speedup. - Fair generation timing: Describe how the notebook compares speculative generation with T alone and checks that the outputs agree.
Why it matters: A controlled comparison distinguishes a real generation benefit from timing noise and startup overhead. - Measured capstone speedup: State the observed acceptance, speedup, and output agreement, and relate the modest gain to the draft's cost.
Why it matters: The result shows that near-perfect acceptance can still yield only a modest practical improvement.
Key terms: speculative decoding, draft model, target model, greedy decoding, acceptance rate, α, c, draft step, structural tokens, timed_generate
13.6.2 Cascaded routing with structural verification
This section implements a small-model-first cascade that routes structured tool calls using syntax and confidence checks, then evaluates its quality and latency trade-offs.
- Cascade routing criteria: Trace how the router decides whether to return model S's tool call or regenerate it with model M or T.
Why it matters: These checks let the smaller model serve suitable requests without invoking the larger model. - Weakest token confidence: Explain how the router measures S's confidence and why it uses the least confident token rather than an average.
Why it matters: A single uncertain detail can undermine an otherwise confident structured call. - Cascade latency trade-off: Explain the cascade's approximate latency and how its use of the larger model differs from speculative decoding.
Why it matters: The cascade can lower average cost and latency, but fallback requests pay for two generations. - Threshold evaluation trade-offs: Describe what the test results show about the trade-off between exact-match quality, escalation, and latency as the confidence threshold rises.
Why it matters: The experiment shows that matching the larger model's accuracy can erase the latency advantage.
Key terms: cascaded routing, parse_tool_call, CONF_THRESHOLD, target_model, output_logits, min_prob, weakest_token, mean latency, escalation, speculative decoding
13.7 Hands-on lab
13.8 Summary
Concepts explained in this chapter
- speculative decodingSpeculative decoding is a generation technique in which a smaller model proposes tokens that a larger model verifies, preserving the larger model’s output while potentially accelerating generation.
- stratified splitA stratified split divides data into subsets while preserving the representation of relevant categories in each subset.
- cascaded routingCascaded routing is a model-serving strategy that tries a less costly model first and escalates requests that fail specified criteria to a larger model.
- model familyA model family is a set of related models arranged or designed to provide different capabilities or resource requirements.
- chat templateA chat template is a formatting specification that structures messages and other content into a model's expected conversational input format.
- data-driven methodData-driven pruning selects model components for removal using measurements from representative input data.
- expansion ratioAn expansion rate is the ratio of a module's intermediate dimension to the model's hidden-state dimension, expressed as a percentage or factor.
- hybrid modeHybrid mode combines neuron weight information with activation statistics measured on calibration inputs when ranking neurons for pruning.
- peak-to-peak magnitude pruningPeak-to-peak magnitude pruning ranks neurons using the peak-to-peak magnitude of their activations or weights.
- distillationKnowledge distillation is a model-compression technique in which a smaller student model learns to reproduce useful behavior from a larger teacher model.
- loss maskingLabel masking excludes selected token positions from contributing to a model's training loss while leaving the input available to the model.
- exact-match accuracyExact match is a metric that counts outputs as correct only when they exactly match the reference under the specified comparison rules.
- draft modelA draft model proposes candidate tokens for a target model to verify during speculative decoding.
- target modelA target model verifies candidate tokens proposed by a draft model during speculative decoding.
- confidence thresholdA confidence threshold is a cutoff applied to a model's token probability to decide whether its output is sufficiently reliable.
- baselineA quantitative baseline is an objective reference measurement established before interventions to evaluate their effects.
- benchmarkA benchmark is a standardized task or test suite used to measure and compare model performance.
- fine-tuningFine-tuning that adapts a model for a particular specialization or domain.
- low-rank adaptationLow-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that trains low-rank updates while keeping the original model weights frozen.
- custom modelA specialized model is a machine-learning model adapted for a particular domain, dataset, or set of tasks and assigned as an expert within a larger model.
- paddingPadding is the addition of extra elements to a computational dimension so its size meets a hardware or algorithmic alignment requirement.
- gradient accumulationGradient accumulation is a training technique that sums gradients across multiple batches before updating model parameters.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
