liveBook · Manning

Routing

As explained in Rearchitecting LLMs

Routing is the process of directing requests to a selected model or service according to a system’s rules.

In an agent pipeline, routing can direct rigid JSON-generation requests from a costly external frontier API to a self-hosted specialized small language model, with an API fallback. section 12.0

Reduce the risk of replacing the established API by shadowing live traffic: test the candidate model while users continue receiving API responses, then validate cost, latency, and reliability. section 12.0

How Rearchitecting LLMs covers it

How deep it goes
Frames a safe routing strategy around shadow validation but does not specify routing conditions or implementation steps.
Examples
Uses a four-line prompt for typed JSON arguments and shadow-tests a candidate model against live agent traffic.
What it assumes
Assumes an agent pipeline that calls an external API to generate typed tool arguments.
What you can do after
Evaluate a self-hosted model replacement against cost, latency, and reliability while users continue receiving API responses.
How it connects
Connects routing to agent pipelines, small language models, API fallbacks, and shadow testing.

Also explained in

Compare how 34 books explain routing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.