A better model score does not always mean a better system. Human-in-the-loop evaluation asks people to judge outputs or weigh improvements against regressions when automated metrics cannot settle the decision. section 13.2 section 12.1.4
Choose the human judgment that matches the decision:
- For search quality, ask assessors to rate results for benchmark queries or choose between old and new results. Pairwise choices tend to be more robust than subjective numerical scores. section 12.1.4
- For an offline search metric, collect judgments about whether results look reasonable. This requires additional data, unlike a diversity count that can be calculated without new labels. section 5.3.2
- For a release, have a responsible person decide whether the gains justify new errors. That reviewer may be an ML engineer, product manager, domain expert, or a crowd of users. section 13.2
A release with mixed results
Suppose an updated model fixes three errors in a 100-sample test set but creates two new ones. Its aggregate performance improves, yet someone still needs to judge whether the new failures are acceptable before release. section 13.2
Human evaluation can be precise and trustworthy, but assessor time, cost, delay, and ambiguity limit its throughput. Use it where judgment matters most; it is not a substitute for direct online testing on real data. section 12.1.4
