Confidence is a numerical signal for judging how strongly a language model supports its response. Token log probabilities expose this signal, helping an application assess reliability and choose a response strategy. section 7.2.6
A production-oriented confidence workflow can:
- Request per-token log probabilities and the most probable alternatives. section 7.2.6
- Inspect an initial token’s probability as an indication of whether the model has enough information. section 7.2.6
- Apply a confidence threshold and ask the user for clarification when confidence is insufficient. section 7.2.6
Threshold-based response handling
Configure ChatOpenAI with logprobs enabled and request the top three alternatives, then pass a confidenceThreshold of 85 to an enhanced RAG query and use ask_user as the fallback. section 7.2.6
Figure 7.10 RAG approach for grounding LLMs with relevant information. The system performs a grounding search to extract the most relevant context, which is combined with the original user input to create an enhanced prompt. This prompt is then passed to the LLM, which generates the final grounded response that is provided to the user.
Follow the grounding flow from retrieval, through context-enriched prompting, to a response whose reliability can be assessed. section 7.2.6
