12 Case study — replacing one agent call with a specialized SLM in production
Guest chapter by Ishita Verma
This chapter covers
- Identifying an agent call worth replacing
- Turning agent traffic into a pruned, distilled SLM
- Serving a non-standard model with vLLM
- Aligning pruned widths and sizing the KV cache
- Routing with an API fallback and proving it pays off
Late one evening during a major feature rollout, our team watched an agent pipeline’s p99 latency curve march steadily upward under live user traffic. On paper, the architecture was efficient: an autonomous loop that parsed intent, fetched context, and chained internal tools. But in practice, a single step inside the loop was stalling the entire engine. Every few seconds, the agent paused, handed a four-line prompt to an external frontier API, and waited for a small, rigid JSON payload containing tool arguments before it could take its next step.
When traffic spiked, the network round-trips, multi-tenant provider queues, and per-token bills quickly compounded. What should have been a snappy, interactive experience felt sluggish and unpredictable, held hostage by an external endpoint with rate limits we couldn't control. The paradox was hard to ignore: we were paying for the massive reasoning capacity of a frontier foundation model on every iteration, merely to parse a handful of typed arguments.