liveBook · Manning

Smooth scorer

As explained in Self-Improving Agents

A scoring function that awards graded partial credit according to how close an answer is to the desired result.

An exact-match check gives an answer 1.00 for a match or 0.00 for a miss. That is a useful verdict, but a poor guide for search: fixing part of an answer can still leave the score at zero. A smooth scorer awards graded partial credit, making progress visible. section 3.2.2

A leaderboard that is wrong in different ways

A leaderboard with inflated totals scores 0.00 under exact match. A smooth scorer can check the names, their order, and how far each total is from the truth. The same answer scores 0.63: still wrong, but distinguishable from one that gets every part wrong. section 3.2.2

To make the score useful for an improvement loop:

  • Choose how much credit correct names, correct order, and close totals earn. section 3.5
  • Check the checker: the known-correct oracle answer must score 1.00 before the score is trusted. section 3.2.2 section 3.5
  • Correct wrong totals one at a time and confirm that each fix raises the score. Look for answers that could collect partial credit without being right. section 3.5

Keep the two jobs separate: use strict exact match to validate the checker, and the smooth score to guide search. A gate needs a reliable verdict; a search needs a slope it can climb. section 3.2.2

How Self-Improving Agents covers it

How deep it goes
Defines partial-credit scoring and gives design and validation checks, but leaves the component weights to the implementer.
Examples
Compares 0.00 exact match with 0.63 partial credit on a wrong leaderboard, then tests one-total-at-a-time corrections.
What it assumes
Assumes basic Python and an existing agent checker; teaches the distinction between strict and smooth scores from zero.
What you can do after
Build it from scratch by weighting answer components, validating the oracle score, and checking that incremental fixes raise the score.
How it connects
Connects exact-match checking, ground truth, oracle validation, and search.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Self-Improving Agents. The section links point to the parts of the book it draws on.