liveBook · Manning

Chapter 12: Case study — replacing one agent call with a specialized SLM in production

What you will learn in chapter 12 of Rearchitecting LLMs

The chapter introduces a costly agent-loop bottleneck and the use of shadow testing to reduce the risk of replacing its frontier API with a specialized SLM.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduces a costly agent-loop bottleneck and the use of shadow testing to reduce the risk of replacing its frontier API with a specialized SLM.

Key terms: agent loop, frontier API, tool arguments, JSON payload, shadow testing, live traffic, candidate small language model

12.1 One call in the agent's loop

This section explains why a narrow, frequent, blocking agent call is a candidate for replacement by a specialized SLM trained against the frontier model.

Key terms: narrow, frequent, blocking, critical path, frontier API, specialized SLM, structured pruning, frontier model, teacher, bar

12.2 Turning agent traffic into a specialization set

This section turns real agent calls into a filtered specialization dataset and illustrates its construction for tool argument extraction.

Key terms: specialization set, frontier API, input distribution, teacher-labeled, schema-valid, downstream confirmation, confirmed pairs, live traffic, synthetic data, teacher

12.3 From baseline to specialized SLM

This section establishes accuracy baselines, structurally specializes a model for tool argument extraction, and recovers its task performance through sequence-level distillation.

Key terms: base model, performance baseline, frontier API, capabilities harness, structured tool argument extraction, depth pruning, width pruning, hardware-tile multiple, selective attention removal, KV cache

12.4 The research result, and what production has to fix

This section reports the specialized SLM’s research result and explains why its modified architecture complicates production serving.

Key terms: held-out slice, accuracy, parameters, frontier API, research result, production, batched offline data, serving live, depth, non-standard widths

12.5 The serving problem for a rearchitected model

This section distinguishes the serving consequences of depth pruning, width pruning, and attention removal, and identifies the model topology used in the case study.

Key terms: depth pruning, transformer blocks, config, cache, width pruning, feed-forward intermediate size, padding, heterogeneous shapes, attention removal, model definition

12.6 Will it load: from research artifact to servable model

This section explains why a rearchitected model may need custom serving support and how its pruning pattern determines the loading approach.

Key terms: Transformers, eager mode, vLLM, TensorRT-LLM, depth pruning, width pruning, uniform and aligned, heterogeneous per layer, custom registration, fallback

12.6.1 What the checkpoint contains

This section inspects how an attention-pruned checkpoint records its non-standard block structure and why that matters for serving.

Key terms: LlamaForCausalLM, dropped_attn_layers, checkpoint, non-standard structure, q_proj, k_proj, v_proj, o_proj, input_layernorm, MLP

12.6.2 Loading the unmodified model

Establishes a vLLM loading, KV-cache, and generation-speed baseline for the unmodified model before testing the pruned model.

Key terms: Llama-3.2-3B, KV cache, batch size 1, time per token, vllm_probe.py, run_vllm, subprocess, log file, eager execution, bytes per token

12.6.3 Loading a model with non-standard architecture

This section shows how to load an attention-pruned Llama checkpoint in vLLM using a custom architecture plugin and interpret the resulting cache capacity.

Key terms: LlamaForCausalLM, input_layernorm, attention module, dropped_attn_layers, post_attention_layernorm, MLP, residual, KV cache, layer_type, register

12.7 Is it efficient: alignment and the KV cache

This introduction connects pruned-model efficiency on real hardware to GPU alignment and previews two serving calculations.

Key terms: width pruning, GPU architecture, padding

12.7.1 The padding tax

This section explains how GPU tile alignment affects the compute savings realized from width pruning and how to choose an aligned target.

Key terms: tiles, padding, tensor cores, intermediate dimension, aligned width, tile boundary, pruning ratio, padding tax

12.7.2 KV cache arithmetic after pruning

This section explains how pruning reduces KV cache requirements, illustrates the savings, and cautions that serving engines must realize them in practice.

Key terms: key-value cache, L_attn, KV heads, head dimension, sequence length, batch size, attention blocks, servable batch, decode-bound workload, throughput

12.8 Is it reliable: serving in the agent loop

This section explains why serving an agent’s repeated calls requires latency-focused configuration rather than throughput optimization alone.

Key terms: decode-bound, latency-sensitive, time to first token, per-token decode speed, small batch, batching window, throughput, latency

12.8.1 Making the output valid by construction

This section shows how schema-constrained decoding makes an SLM’s agent-facing output structurally valid while leaving its content quality to the model.

Key terms: structured decoding, schema, structured-output parameters, valid by construction, tool call, anyOf, search, sampling parameters, plugin overrides, vLLM

12.8.2 A safety net for the SLM in production

This section explains confidence-based API fallback and a staged rollout that safeguards production use of a specialized SLM.

Key terms: router pattern, uncertainty, least confident token, escalation rate, threshold, shadow, disagreement rate, critical path, canary

12.9 Proving self-hosting pays off

This section evaluates whether a self-hosted SLM pays off by comparing its cost, latency, and task quality with an API in an illustrative agent deployment.

Key terms: break-even request rate, cost per request, utilization, escalation, p50, p95, p99, latency, task quality, API baseline

12.10 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 11: Precision pruning for bias Chapter 13: Capstone — cascading specialists into a model family for agents →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.