liveBook · Manning

Confidence threshold

As explained in Rearchitecting LLMs

A confidence threshold is a cutoff applied to a model's token probability to decide whether its output is sufficiently reliable.

In cascaded tool-call routing, a confidence threshold helps decide whether to return a smaller model’s call or escalate the same prompt to a larger model. Confidence is measured by the lowest probability among the generated tokens, not their average: one uncertain token can undermine a function name or argument. section 13.6.2

Apply the cutoff only after checking the call’s structure:

  • Escalate immediately if the output is not valid JSON or lacks a function name. section 13.6.2
  • Return the smaller model’s call when it is valid and its minimum token probability meets or exceeds CONF_THRESHOLD; otherwise send the same prompt to the target model. section 13.6.2
  • Keep the minimum probability and weakest token with the result so escalations can be inspected. section 13.6.2

Threshold behavior

With a 0.90 cutoff, a ZIP-code request with minimum probability 0.999 is served by the smaller model. A New York forecast request at 0.547 is escalated and answered correctly by the target model. A Los Angeles request at 0.951 stays with the smaller model but does not match the reference call, showing that passing the cutoff does not guarantee an exact match. section 13.6.2

Figure 13.6 Cascaded routing with structural verification. Every request goes first to S, which generates the tool call and records the lowest probability among its generated tokens. If the call parses into valid JSON with a function name and that minimum probability reaches the threshold, S's call is returned. Otherwise the same prompt is sent to the target model T.

The routing flow shows how syntax validation and minimum-token confidence gate whether a request stays with the smaller model or is escalated. section 13.6.2

How Rearchitecting LLMs covers it

How deep it goes
Computes minimum generated-token probability and combines a cutoff with JSON and function-name checks.
Examples
Tests ZIP-code and weather calls, including a 0.547 New York forecast escalated to the target model.
What it assumes
Assumes generated tool calls can be parsed as JSON and checked for a function name.
What you can do after
Configure CONF_THRESHOLD to gate valid calls from a smaller model and escalate calls that miss the cutoff.
How it connects
Combines token probabilities with JSON validation, tool calls, and cascaded model routing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.