A model update can improve an overall score while introducing new errors. Human-in-the-loop evaluation brings people’s judgments into decisions about output quality and whether the tradeoffs justify a release. section 13.2 section 12.1.4
Before deployment, assessors can rate outputs or compare alternatives. Direct comparisons tend to be more robust than numerical ratings because they avoid subjective differences in score estimates. Human evaluation can be precise and trustworthy, but limited throughput, ambiguity, delays, and cost make frequent use difficult. section 12.1.4
Evaluating search results
For benchmark queries, experts can rate search relevance from 1 to 5 or choose between results from the old and new pipelines. For an image search service, another measure is the proportion of users who consider the results reasonable. Collecting these judgments takes time and effort, unlike an offline diversity measure that needs no additional data. section 12.1.4 section 5.3.2
Choose human involvement to fit the decision:
- Use reviewers to weigh improvements against regressions before release. Depending on the system, reviewers might be ML engineers, product managers, domain experts, or crowdsourced users. section 13.2
- For generated language, consider human evaluation or a hybrid approach using a critic model trained on human feedback. section 12.1.4
- When automatic model updates are necessary for scale, use especially thorough testing and disable updates if malfunction becomes possible. section 13.2



























