Chapter 12: Case study — replacing one agent call with a specialized SLM in production
The chapter introduces a costly agent-loop bottleneck and the use of shadow testing to reduce the risk of replacing its frontier API with a specialized SLM.
After this chapter you will be able to
- Agent call replacement rationale. Explain why the external frontier API call was a candidate for replacement in the agent loop.
- Shadow testing replacement risk. Describe how shadow testing reduced the risk of replacing the established API.
- Agent Call Replacement Rationale. Explain how the call's narrow, frequent, and blocking nature makes it both costly and replaceable.
- Specialized SLM Through Pruning. Explain why training and structured pruning for one job can produce a smaller specialized SLM.
- Frontier Model Teacher Role. Explain why the frontier model can serve as both teacher and performance bar for the specialized SLM.
- Live traffic training pairs. Explain how logged agent calls become specialization examples and why live prompts are valuable.
- Validity and execution filters. Describe how responses are filtered and how downstream confirmation is recorded and used.
- Task-Specific Structural Edits. Explain the three coordinated structural edits and why their importance is measured on the specialization set.
- Sequence-Level Performance Recovery. Explain why the pruned model needs recovery training, why this case uses teacher completions rather than soft-label logit distillation, and what accuracy it achieves.
- Held-out specialization result. Describe how the specialized SLM compares with the frontier API on the held-out sub-task.
- Research versus production evidence. Distinguish the offline research result from the live serving benefit the agent needs.
- Serving architecture mismatch. Explain how the model’s structural modifications create challenges for stock high-throughput serving engines.
- Depth pruning serving consequences. Explain why a depth-pruned model is relatively straightforward to serve and what it saves.
- Width pruning serving costs. Explain how width pruning can cause padding waste and unexpected layer shapes in a serving engine.
- Attention removal serving disruption. Explain why removing attention from only some blocks affects both model loading and cache management.
- Research and serving gap. Explain why a model that runs in Transformers may not immediately run in a fast production serving engine.
- Pruning pattern loading choices. Distinguish the loading approaches for depth pruning, uniform aligned width pruning, and heterogeneous width or missing attention.
- Checkpoint structure and serving. Explain how the checkpoint records attention pruning and why its non-standard structure matters when loading it.
- Unmodified model serving baseline. Explain why the original Llama-3.2-3B is loaded first and what the run establishes for comparison.
- Isolated vLLM probe workflow. Describe how the probe and wrapper load and report on a model, and explain why each load runs in a separate process.
- Baseline KV cache arithmetic. Explain how the run's reserved KV-cache memory and token capacity are checked against the model's predicted bytes per token.
- Standard Architecture Loading Failure. Explain why vLLM cannot load the attention-pruned checkpoint using its declared standard Llama architecture.
- Attention-Free Decoder Layer. Describe how the custom decoder layer handles standard and attention-free blocks, including its residual computation.
- Plugin Registration and Overrides. Trace how the custom layer is connected to Llama and made available to vLLM when loading the checkpoint.
- Cache Capacity and Results. Explain the observed KV cache savings and distinguish the effects of lower per-token cost and available cache memory.
- Width Pruning GPU Alignment. Explain why width pruning should account for GPU architecture when assessing a model’s efficiency.
- GPU Tile Padding Costs. Explain why an unaligned feed-forward width can cost more GPU compute than its size suggests.
- Aligned Pruning Target Selection. Apply the section's rounding approach to distinguish the width a kernel serves from an aligned pruning target, using the 20 percent cut as an example.
- KV cache scaling factors. Explain how KV cache size scales with the number of attention-bearing blocks, KV heads, head dimension, sequence length, and batch size, and how pruning changes it.
- Pruned cache workload example. Describe the cache comparison between the original and reference topologies and explain what the saving means for different workloads.
- Paged attention realization caveat. Explain why predicted cache savings must be checked with the serving engine and what the Transformers fallback provides.
- Agent workload and latency. Explain how the agent’s call pattern determines the serving metrics that matter.
- Schema constrained decoding. Explain why structured decoding is needed for the replacement agent call and how it constrains generation.
- Structural validity versus quality. Distinguish the structural guarantee of constrained decoding from the quality of the model’s chosen content.
- Confidence Based API Fallback. Explain how the router uses SLM confidence to decide whether to serve its answer or fall back to the original API, and how the threshold balances quality and cost.
- Shadow and Canary Rollout. Describe how shadow deployment and a canary introduce the SLM safely into the agent's request path.
- Self-hosting cost and utilization. Explain how request volume and API escalation affect the cost of self-hosting, and apply the section’s break-even and per-request calculations.
- Latency and quality comparison. Describe how the section compares SLM and API latency and task quality, and distinguish its illustrative figures from measurements.
- Fallback and routed tail. Explain why the routed system’s slow tail can differ from the latency of the SLM path alone.
What it builds on
- How depth pruning works (chapter 4) · The earlier explanation of how depth pruning removes model structure.
- Recovering the knowledge (chapter 6) · The earlier treatment of recovering knowledge through teacher-student training.
- Identifying neurons for removal (chapter 5) · Removing neurons from the feed-forward expansion layers.
- Structural implications and framework compatibility (chapter 8) · Structural implications and framework compatibility of removing attention modules.
- Reconstructing the MLP module (chapter 5) · Width pruning changes the MLP shape.
- What the KV cache is and why it matters (chapter 8) · The role of the KV cache in inference.
- Selecting static neurons (chapter 5) · The feed-forward width can be reduced by removing neurons.
Section by section
Introduction
The chapter introduces a costly agent-loop bottleneck and the use of shadow testing to reduce the risk of replacing its frontier API with a specialized SLM.
- Agent call replacement rationale: Explain why the external frontier API call was a candidate for replacement in the agent loop.
Why it matters: Identifying a repeated, constrained bottleneck gives the team a concrete target for specialization. - Shadow testing replacement risk: Describe how shadow testing reduced the risk of replacing the established API.
Why it matters: Shadow testing lets a team evaluate a replacement without immediately changing what users receive.
Key terms: agent loop, frontier API, tool arguments, JSON payload, shadow testing, live traffic, candidate small language model
12.1 One call in the agent's loop
This section explains why a narrow, frequent, blocking agent call is a candidate for replacement by a specialized SLM trained against the frontier model.
- Agent Call Replacement Rationale: Explain how the call's narrow, frequent, and blocking nature makes it both costly and replaceable.
Why it matters: These properties explain why replacing this particular call can improve the agent's economics and responsiveness. - Specialized SLM Through Pruning: Explain why training and structured pruning for one job can produce a smaller specialized SLM.
Why it matters: Focusing on one job makes it possible to remove model capacity that the agent's sub-task does not need. - Frontier Model Teacher Role: Explain why the frontier model can serve as both teacher and performance bar for the specialized SLM.
Why it matters: The model being replaced supplies both a source of task behavior and a reference for judging its replacement.
Key terms: narrow, frequent, blocking, critical path, frontier API, specialized SLM, structured pruning, frontier model, teacher, bar
12.2 Turning agent traffic into a specialization set
This section turns real agent calls into a filtered specialization dataset and illustrates its construction for tool argument extraction.
- Live traffic training pairs: Explain how logged agent calls become specialization examples and why live prompts are valuable.
Why it matters: Real agent traffic provides examples matched to the specific job the smaller model must perform. - Validity and execution filters: Describe how responses are filtered and how downstream confirmation is recorded and used.
Why it matters: Filtering keeps malformed teacher behavior out and favors examples that actually worked. - Traffic volume and supplementation: Explain how traffic volume affects the need for synthetic supplementation.
Why it matters: The available traffic determines whether collecting real examples is enough. - Sensitive prompt logging: Explain when and how to protect sensitive prompt data during collection.
Why it matters: A privacy policy applied at logging time prevents sensitive prompt data from being written unprotected. - Tool extraction reference dataset: Describe the case study's sub-task and how its illustrative traffic sample became training and evaluation sets.
Why it matters: The example connects the collection and filtering rules to a concrete train-and-evaluate dataset.
Key terms: specialization set, frontier API, input distribution, teacher-labeled, schema-valid, downstream confirmation, confirmed pairs, live traffic, synthetic data, teacher
12.3 From baseline to specialized SLM
This section establishes accuracy baselines, structurally specializes a model for tool argument extraction, and recovers its task performance through sequence-level distillation.
- Model and Accuracy Baselines: Describe the base model choice and the two accuracy baselines used for the specialization task.
Why it matters: These measurements show where the starting model stands and what performance the specialist must approach. - Task-Specific Structural Edits: Explain the three coordinated structural edits and why their importance is measured on the specialization set.
Why it matters: Task-specific pruning reduces the model while targeting structures less important to the actual agent sub-task. - Sequence-Level Performance Recovery: Explain why the pruned model needs recovery training, why this case uses teacher completions rather than soft-label logit distillation, and what accuracy it achieves.
Why it matters: Recovery can restore task accuracy after structural pruning even when the API does not provide full token distributions.
Key terms: base model, performance baseline, frontier API, capabilities harness, structured tool argument extraction, depth pruning, width pruning, hardware-tile multiple, selective attention removal, KV cache
12.4 The research result, and what production has to fix
This section reports the specialized SLM’s research result and explains why its modified architecture complicates production serving.
- Held-out specialization result: Describe how the specialized SLM compares with the frontier API on the held-out sub-task.
Why it matters: This result establishes the quality and size trade-off before production deployment is considered. - Research versus production evidence: Distinguish the offline research result from the live serving benefit the agent needs.
Why it matters: A promising offline model is not yet evidence that replacing an agent call pays off in production. - Serving architecture mismatch: Explain how the model’s structural modifications create challenges for stock high-throughput serving engines.
Why it matters: The changes that made the model smaller also make it harder to serve efficiently with a stock engine.
Key terms: held-out slice, accuracy, parameters, frontier API, research result, production, batched offline data, serving live, depth, non-standard widths
12.5 The serving problem for a rearchitected model
This section distinguishes the serving consequences of depth pruning, width pruning, and attention removal, and identifies the model topology used in the case study.
- Depth pruning serving consequences: Explain why a depth-pruned model is relatively straightforward to serve and what it saves.
Why it matters: Whole-block removal has different serving implications from changes within blocks. - Width pruning serving costs: Explain how width pruning can cause padding waste and unexpected layer shapes in a serving engine.
Why it matters: A narrower model does not automatically deliver all the expected compute savings or fit an engine's model definition. - Attention removal serving disruption: Explain why removing attention from only some blocks affects both model loading and cache management.
Why it matters: Attention removal creates both a loading problem and a separate cache-allocation compatibility question. - Reference and served topologies: Distinguish the chapter's reference model design from the checkpoint actually served.
Why it matters: Keeping the two designs distinct prevents attributing modifications to the checkpoint that is actually served.
Key terms: depth pruning, transformer blocks, config, cache, width pruning, feed-forward intermediate size, padding, heterogeneous shapes, attention removal, model definition
12.6 Will it load: from research artifact to servable model
This section explains why a rearchitected model may need custom serving support and how its pruning pattern determines the loading approach.
- Research and serving gap: Explain why a model that runs in Transformers may not immediately run in a fast production serving engine.
Why it matters: This distinction explains why a working research model is not automatically ready for fast production serving. - Pruning pattern loading choices: Distinguish the loading approaches for depth pruning, uniform aligned width pruning, and heterogeneous width or missing attention.
Why it matters: Matching the loading approach to the pruning pattern avoids unnecessary custom code and unsupported config-only fixes.
Key terms: Transformers, eager mode, vLLM, TensorRT-LLM, depth pruning, width pruning, uniform and aligned, heterogeneous per layer, custom registration, fallback
12.6.1 What the checkpoint contains
This section inspects how an attention-pruned checkpoint records its non-standard block structure and why that matters for serving.
- Checkpoint structure and serving: Explain how the checkpoint records attention pruning and why its non-standard structure matters when loading it.
Why it matters: A checkpoint can be internally consistent yet require special handling to serve the intended model. - Intact and pruned tensors: Describe how comparing an intact block with an attention-free block confirms what pruning removed.
Why it matters: Comparing saved tensors verifies that the checkpoint's recorded pruning matches its weights.
Key terms: LlamaForCausalLM, dropped_attn_layers, checkpoint, non-standard structure, q_proj, k_proj, v_proj, o_proj, input_layernorm, MLP
12.6.2 Loading the unmodified model
Establishes a vLLM loading, KV-cache, and generation-speed baseline for the unmodified model before testing the pruned model.
- Unmodified model serving baseline: Explain why the original Llama-3.2-3B is loaded first and what the run establishes for comparison.
Why it matters: The original model gives you a concrete reference for judging the pruned model. - Isolated vLLM probe workflow: Describe how the probe and wrapper load and report on a model, and explain why each load runs in a separate process.
Why it matters: The isolated probe makes model-load comparisons repeatable and failures easier to diagnose. - Baseline KV cache arithmetic: Explain how the run's reserved KV-cache memory and token capacity are checked against the model's predicted bytes per token.
Why it matters: This check connects the serving output to the model's expected cache footprint.
Key terms: Llama-3.2-3B, KV cache, batch size 1, time per token, vllm_probe.py, run_vllm, subprocess, log file, eager execution, bytes per token
12.6.3 Loading a model with non-standard architecture
This section shows how to load an attention-pruned Llama checkpoint in vLLM using a custom architecture plugin and interpret the resulting cache capacity.
- Standard Architecture Loading Failure: Explain why vLLM cannot load the attention-pruned checkpoint using its declared standard Llama architecture.
Why it matters: The loading error identifies an architecture mismatch, not simply a bad model identifier. - Attention-Free Decoder Layer: Describe how the custom decoder layer handles standard and attention-free blocks, including its residual computation.
Why it matters: The custom layer makes the engine's computation and allocated components match the pruned checkpoint. - Plugin Registration and Overrides: Trace how the custom layer is connected to Llama and made available to vLLM when loading the checkpoint.
Why it matters: Registering a custom implementation is not enough unless the engine can discover it and the checkpoint is directed to use it. - Cache Capacity and Results: Explain the observed KV cache savings and distinguish the effects of lower per-token cost and available cache memory.
Why it matters: Separating per-token savings from the memory budget prevents overstating what pruning alone achieved. - Alternative Serving Paths: Distinguish ahead-of-time compilation from a custom Transformers-based serving path for this model.
Why it matters: A workable serving choice depends on both architecture stability and the need for a correct fallback.
Key terms: LlamaForCausalLM, input_layernorm, attention module, dropped_attn_layers, post_attention_layernorm, MLP, residual, KV cache, layer_type, register
12.7 Is it efficient: alignment and the KV cache
This introduction connects pruned-model efficiency on real hardware to GPU alignment and previews two serving calculations.
- Width Pruning GPU Alignment: Explain why width pruning should account for GPU architecture when assessing a model’s efficiency.
Why it matters: A smaller model’s efficiency depends on how its dimensions fit the hardware that serves it.
Key terms: width pruning, GPU architecture, padding
12.7.1 The padding tax
This section explains how GPU tile alignment affects the compute savings realized from width pruning and how to choose an aligned target.
- GPU Tile Padding Costs: Explain why an unaligned feed-forward width can cost more GPU compute than its size suggests.
Why it matters: Padding can quietly erode the serving savings expected from pruning. - Aligned Pruning Target Selection: Apply the section's rounding approach to distinguish the width a kernel serves from an aligned pruning target, using the 20 percent cut as an example.
Why it matters: Choosing alignment during pruning makes the removed width translate into actual compute savings.
Key terms: tiles, padding, tensor cores, intermediate dimension, aligned width, tile boundary, pruning ratio, padding tax
12.7.2 KV cache arithmetic after pruning
This section explains how pruning reduces KV cache requirements, illustrates the savings, and cautions that serving engines must realize them in practice.
- KV cache scaling factors: Explain how KV cache size scales with the number of attention-bearing blocks, KV heads, head dimension, sequence length, and batch size, and how pruning changes it.
Why it matters: The cache arithmetic shows which architectural changes can free memory for serving more requests. - Pruned cache workload example: Describe the cache comparison between the original and reference topologies and explain what the saving means for different workloads.
Why it matters: A proportional memory saving only translates into a serving benefit when that memory is a meaningful workload constraint. - Paged attention realization caveat: Explain why predicted cache savings must be checked with the serving engine and what the Transformers fallback provides.
Why it matters: Theoretical cache savings are useful only if the actual serving engine realizes them while serving the model correctly.
Key terms: key-value cache, L_attn, KV heads, head dimension, sequence length, batch size, attention blocks, servable batch, decode-bound workload, throughput
12.8 Is it reliable: serving in the agent loop
This section explains why serving an agent’s repeated calls requires latency-focused configuration rather than throughput optimization alone.
- Agent workload and latency: Explain how the agent’s call pattern determines the serving metrics that matter.
Why it matters: The agent’s progress depends on individual calls returning promptly, not just on maximizing aggregate throughput. - Latency focused serving choices: Describe how the agent’s latency-sensitive workload changes hardware and batching choices.
Why it matters: A configuration that improves aggregate throughput can make the agent slower to respond.
Key terms: decode-bound, latency-sensitive, time to first token, per-token decode speed, small batch, batching window, throughput, latency
12.8.1 Making the output valid by construction
This section shows how schema-constrained decoding makes an SLM’s agent-facing output structurally valid while leaving its content quality to the model.
- Schema constrained decoding: Explain why structured decoding is needed for the replacement agent call and how it constrains generation.
Why it matters: A replacement must produce something the agent can parse before the agent can act on it. - Structural validity versus quality: Distinguish the structural guarantee of constrained decoding from the quality of the model’s chosen content.
Why it matters: Valid JSON removes a parsing failure without proving that the model made the right decision. - Structured output probe setup: Describe how the example configures and runs the structured-output probe.
Why it matters: The probe shows how to apply the schema constraint in the serving setup.
Key terms: structured decoding, schema, structured-output parameters, valid by construction, tool call, anyOf, search, sampling parameters, plugin overrides, vLLM
12.8.2 A safety net for the SLM in production
This section explains confidence-based API fallback and a staged rollout that safeguards production use of a specialized SLM.
- Confidence Based API Fallback: Explain how the router uses SLM confidence to decide whether to serve its answer or fall back to the original API, and how the threshold balances quality and cost.
Why it matters: Fallback lets the system use the cheaper specialist without relying on it for every difficult case. - Shadow and Canary Rollout: Describe how shadow deployment and a canary introduce the SLM safely into the agent's request path.
Why it matters: A staged rollout reveals disagreement on live traffic before the specialist affects most users.
Key terms: router pattern, uncertainty, least confident token, escalation rate, threshold, shadow, disagreement rate, critical path, canary
12.9 Proving self-hosting pays off
This section evaluates whether a self-hosted SLM pays off by comparing its cost, latency, and task quality with an API in an illustrative agent deployment.
- Self-hosting cost and utilization: Explain how request volume and API escalation affect the cost of self-hosting, and apply the section’s break-even and per-request calculations.
Why it matters: The economics depend on real traffic and fallback costs, not just the price of a GPU. - Latency and quality comparison: Describe how the section compares SLM and API latency and task quality, and distinguish its illustrative figures from measurements.
Why it matters: A cost saving is not enough if latency tails or task quality make the replacement unsuitable. - Fallback and routed tail: Explain why the routed system’s slow tail can differ from the latency of the SLM path alone.
Why it matters: Fast SLM responses alone do not establish the tail latency of the complete routed system.
Key terms: break-even request rate, cost per request, utilization, escalation, p50, p95, p99, latency, task quality, API baseline
12.10 Summary
Where this is used later
- (chapter intro) (chapter 13) · The role of a model call within an agent's loop.
- Designing the model family (chapter 13) · The role of an individual model call in an agent's loop.
- Cascaded routing with structural verification (chapter 13) · Uses tool-call confidence to build a router.
Concepts explained in this chapter
- custom modelA specialized model is a machine-learning model adapted for a particular domain, dataset, or set of tasks and assigned as an expert within a larger model.
- key-value cacheA key-value cache stores previously computed attention key and value tensors to accelerate autoregressive inference.
- custom architectureA custom architecture is a model structure implemented outside the standard architectures supported by a serving engine.
- specialization setThe specialization set is the example dataset collected to adapt and evaluate the model for structured tool argument extraction.
- structured tool argument extractionStructured tool argument extraction is the example application's task of converting free-form user intent into validated tool names and typed parameters.
- break-even request rateA break-even request rate is the request volume at which the costs of self-hosting and using an external API are equal.
- self-hostingSelf-hosting is operating a model on infrastructure controlled by the organization deploying it rather than relying on an external service.
- width pruningWidth pruning is a structured pruning technique that removes selected neurons or channels to reduce a model's width.
- small language modelA small language model is a language model with comparatively limited parameters and computational requirements, typically optimized for efficient deployment.
- depth pruningDepth pruning is a model-reduction technique that removes complete layers from a neural network, often targeting later layers.
- structural modificationAttention removal is a model modification that removes attention sub-layers from selected transformer blocks.
- PagedAttentionPagedAttention is an attention-memory management technique that stores key-value cache blocks in pages to improve memory utilization during inference.
- paddingPadding is the addition of extra elements to a computational dimension so its size meets a hardware or algorithmic alignment requirement.
- structured pruningStructured pruning is a model-compression technique that removes organized groups of parameters, such as neurons, channels, or layers.
- Llama-3.2-3BLlama-3.2-3B is a language model in Meta's Llama 3.2 family with approximately three billion parameters.
- latencyLatency is elapsed time measured for a language model request or for producing an individual output token.
- vLLMvLLM is an open-source library and inference engine for efficient large language model serving.
- routerA request router is a component that directs incoming requests to an appropriate processing queue or model configuration.
- time to first tokenTime to first token is the elapsed time from a generation request until the model produces its first output token.
- percentileA percentile is a value below which a specified percentage of observations fall in an ordered distribution.
- intermediate sizeAn intermediate size is the dimensionality of a neural module's intermediate representation, such as the expansion dimension of an MLP.
- teacher modelA teacher model is a model whose outputs or behavior guide the training of a student model during knowledge distillation.
- baselineA quantitative baseline is an objective reference measurement established before interventions to evaluate their effects.
- routingRouting is the process of directing requests to a selected model or service according to a system’s rules.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
