chapter twelve

12 Case study — replacing one agent call with a specialized SLM in production

 
Guest chapter by Ishita Verma

This chapter covers

  • Identifying an agent call worth replacing
  • Turning agent traffic into a pruned, distilled SLM
  • Serving a non-standard model with vLLM
  • Aligning pruned widths and sizing the KV cache
  • Routing with an API fallback and proving it pays off

Late one evening during a major feature rollout, our team watched an agent pipeline’s p99 latency curve march steadily upward under live user traffic. On paper, the architecture was efficient: an autonomous loop that parsed intent, fetched context, and chained internal tools. But in practice, a single step inside the loop was stalling the entire engine. Every few seconds, the agent paused, handed a four-line prompt to an external frontier API, and waited for a small, rigid JSON payload containing tool arguments before it could take its next step.

When traffic spiked, the network round-trips, multi-tenant provider queues, and per-token bills quickly compounded. What should have been a snappy, interactive experience felt sluggish and unpredictable, held hostage by an external endpoint with rate limits we couldn't control. The paradox was hard to ignore: we were paying for the massive reasoning capacity of a frontier foundation model on every iteration, merely to parse a handful of typed arguments.

12.1 One call in the agent's loop

12.2 Turning agent traffic into a specialization set

12.3 From baseline to specialized SLM

12.4 The research result, and what production has to fix

12.5 The serving problem for a rearchitected model

12.6 Will it load: from research artifact to servable model

12.6.1 What the checkpoint contains

12.6.2 Loading the unmodified model

12.6.3 Loading a model with non-standard architecture

12.7 Is it efficient: alignment and the KV cache

12.7.1 The padding tax

12.7.2 KV cache arithmetic after pruning

12.8 Is it reliable: serving in the agent loop

12.8.1 Making the output valid by construction

12.8.2 A safety net for the SLM in production

12.9 Proving self-hosting pays off

12.10 Summary