liveBook · Manning

Cascaded routing

As explained in Rearchitecting LLMs

Cascaded routing is a model-serving strategy that tries a less costly model first and escalates requests that fail specified criteria to a larger model.

Cascaded routing reduces serving cost by sending every request to a small model S first. If S produces an acceptable structured tool call, return it; otherwise, send the same prompt to a larger target model, usually T. Unlike speculative decoding, T stays idle for requests S can handle. section 13.6 section 13.6.2

Accept S's output only when both checks pass:

  • Parse the completion as valid JSON containing a function name. section 13.6.2
  • Require the least confident generated token to meet a probability threshold. section 13.6.2

The router can use deterministic checks instead of a classifier or judge model, which would add a call to every request. If S takes fraction c of T's time and fraction p of requests escalate, approximate mean latency is (c + p) times T's. With c near 0.8 and p around 11–13%, the estimate is about 0.9 times T's latency; escalated requests are slower than using T alone. These figures illustrate the method, not expected production gains: the test used small models, 54 examples, and one request at a time on a T4 GPU. section 13.6.2 section 13.6

Figure 13.6 Cascaded routing with structural verification. Every request goes first to S, which generates the tool call and records the lowest probability among its generated tokens. If the call parses into valid JSON with a function name and that minimum probability reaches the threshold, S's call is returned. Otherwise the same prompt is sent to the target model T.

Follow the decision path from S's output through structural and confidence checks to either immediate return or escalation. section 13.6.2

Table 13.4 Three requests through the router
Request
S model min. probability
Weakest token
Served by
Call matches reference
City from a ZIP code
0.999
<tool_call>
S
yes
Forecast for Los Angeles
0.951
,
S
no
Forecast for New York City
0.547
yes
T
yes

Compare three routed requests, including an S-served call that does not match the reference and a low-confidence call escalated to T. section 13.6.2

How Rearchitecting LLMs covers it

How deep it goes
Implements JSON/function-name and minimum-token-probability gates, then estimates latency while cautioning against production extrapolation.
Examples
Compares ZIP-code and city-forecast tool calls, showing S-served outputs and a low-confidence request escalated to T.
What it assumes
Assumes structured tool-call outputs and models from a shared tokenizer, vocabulary, and base lineage.
What you can do after
Operate a two-model router that returns S's output when both checks pass and otherwise regenerates with the target model.
How it connects
Pairs with speculative decoding as a complementary way for related models to cooperate at inference time.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.