liveBook · Manning

Probability-weighted scoring

As explained in Retrieval Augmented Generation, The Foundational Ideas

Probability-weighted scoring aggregates evaluation judgments according to the model's assigned outcome probabilities.

Probability-weighted scoring aggregates evaluation judgments according to the model’s assigned outcome probabilities. In LLM judging, generating evaluation steps before scoring can improve agreement with human judgments. section 14.7

G-Eval judging

G-Eval combines generated evaluation steps with probability-weighted scoring to make automated judgments correlate more closely with human assessments. section 14.7

Use the technique with disciplined evaluation practice:

  • Generate evaluation steps before assigning scores. section 14.7
  • Compare scores across systems and compute intervals before treating differences as regressions. section 14.7

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains generated evaluation steps and probability-weighted scoring, but does not detail the underlying probability calculation.
Examples
Uses G-Eval as the concrete example of probability-weighted LLM judging.
What it assumes
Assumes familiarity with LLM-based evaluation and scoring judgments.
What you can do after
Evaluate RAG systems with judging procedures designed to correlate more closely with human assessments.
How it connects
Connects LLM evaluation with G-Eval, chain-of-thought, human judgments, and confidence intervals.

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.