liveBook · Manning

Confidence

As explained in Building LLM Applications with DSPy

A confidence score is a numerical estimate of how certain a model is in an answer or prediction.

A confidence score is an additional prediction field that expresses how certain a model is about its answer. In DSPy, add `confidence` to the signature; fields default to strings, but a type annotation can request a numeric value. section 2.5.1

"question -> answer, confidence: float"

Use a float annotation when the confidence output should be numeric. section 2.5.1

Requesting confidence

For “What is the capital of France?”, a prediction can return Paris with confidence 1.0, indicating complete confidence. section 2.5.1

Confidence can also affect evaluation scores: reward higher confidence on correct predictions and penalize high confidence on incorrect ones. If ideal confidence is known for an example, compare the prediction with that target; an ambiguous intent might have an ideal confidence of about 80%. section 4.2.1

How Building LLM Applications with DSPy covers it

How deep it goes
Defines confidence as an output field, shows string-to-float typing and one result, then sketches evaluation scoring without formulas or calibration procedures.
Examples
Shows a Paris prediction with confidence 1.0 and an ambiguous flight-intent example with ideal confidence near 80%.
What you can do after
Apply DSPy signatures to request confidence alongside predictions and incorporate it into evaluation scores.
How it connects
Connects confidence to DSPy signatures, classification evaluation, Brier score, and log loss.

Also explained in

Compare how 13 books explain confidence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building LLM Applications with DSPy. The section links point to the parts of the book it draws on.