Cascaded routing reduces serving cost by sending every request to a small model S first. If S produces an acceptable structured tool call, return it; otherwise, send the same prompt to a larger target model, usually T. Unlike speculative decoding, T stays idle for requests S can handle. section 13.6 section 13.6.2
Accept S's output only when both checks pass:
- Parse the completion as valid JSON containing a function name. section 13.6.2
- Require the least confident generated token to meet a probability threshold. section 13.6.2
The router can use deterministic checks instead of a classifier or judge model, which would add a call to every request. If S takes fraction c of T's time and fraction p of requests escalate, approximate mean latency is (c + p) times T's. With c near 0.8 and p around 11–13%, the estimate is about 0.9 times T's latency; escalated requests are slower than using T alone. These figures illustrate the method, not expected production gains: the test used small models, 54 examples, and one request at a time on a T4 GPU. section 13.6.2 section 13.6
Figure 13.6 Cascaded routing with structural verification. Every request goes first to S, which generates the tool call and records the lowest probability among its generated tokens. If the call parses into valid JSON with a function name and that minimum probability reaches the threshold, S's call is returned. Otherwise the same prompt is sent to the target model T.

Follow the decision path from S's output through structural and confidence checks to either immediate return or escalation. section 13.6.2
Table 13.4 Three requests through the router
|
Request
|
S model
min. probability
|
Weakest
token
|
Served by
|
Call
matches reference
|
|
City from a
ZIP code
|
0.999
|
<tool_call>
|
S
|
yes
|
|
Forecast for
Los Angeles
|
0.951
|
,
|
S
|
no
|
|
Forecast for
New York City
|
0.547
|
yes
|
T
|
yes
|
Compare three routed requests, including an S-served call that does not match the reference and a low-confidence call escalated to T. section 13.6.2
