An exact-match check gives an answer 1.00 for a match or 0.00 for a miss. That is a useful verdict, but a poor guide for search: fixing part of an answer can still leave the score at zero. A smooth scorer awards graded partial credit, making progress visible. section 3.2.2
A leaderboard that is wrong in different ways
A leaderboard with inflated totals scores 0.00 under exact match. A smooth scorer can check the names, their order, and how far each total is from the truth. The same answer scores 0.63: still wrong, but distinguishable from one that gets every part wrong. section 3.2.2
To make the score useful for an improvement loop:
- Choose how much credit correct names, correct order, and close totals earn. section 3.5
- Check the checker: the known-correct oracle answer must score 1.00 before the score is trusted. section 3.2.2 section 3.5
- Correct wrong totals one at a time and confirm that each fix raises the score. Look for answers that could collect partial credit without being right. section 3.5
Keep the two jobs separate: use strict exact match to validate the checker, and the smooth score to guide search. A gate needs a reliable verdict; a search needs a slope it can climb. section 3.2.2
