Probability-weighted scoring aggregates evaluation judgments according to the model’s assigned outcome probabilities. In LLM judging, generating evaluation steps before scoring can improve agreement with human judgments. section 14.7
G-Eval judging
G-Eval combines generated evaluation steps with probability-weighted scoring to make automated judgments correlate more closely with human assessments. section 14.7
Use the technique with disciplined evaluation practice:
- Generate evaluation steps before assigning scores. section 14.7
- Compare scores across systems and compute intervals before treating differences as regressions. section 14.7
