An evaluator, or LLM judge, uses an AI model to assess outputs against defined criteria. It can score large numbers of samples quickly, making it useful for model comparisons, stress tests, and production monitoring. Its judgments are signals, not universal substitutes for human review: validate them against high-quality human judgments, and supplement them with human review in high-stakes settings. section 5.4.4
A judge starts with a prompt that embeds a rubric, describes the task, and specifies the output format. For example, a coherence judge can compare a news summary with its source article, check its organization, and assign a score from 1 to 5. The choice of scale matters: numeric scores can be difficult to apply reliably when adjacent values are hard to distinguish. section 5.6
Make the judgment task explicit:
- Specify the analysis steps and the exact permitted output format, such as a fixed JSON object. section 5.6.2
- State which information sources may be used and make policies, missing-evidence rules, and other assumptions explicit. section 5.6.2
- Ask for a categorical decision when hedging would make the result ambiguous. section 5.6.2
Model selection is a trade-off between judgment quality and cost. A strong general-purpose model is a useful starting point for testing the rubric and establishing a performance ceiling. A weaker model may suffice for simple checks, while deep domain expertise may justify a specialized judge. Starting weak can be cheaper, but it can make failures harder to debug when both the prompt and model are weak. section 5.6.1
Judgments can change with prompt wording, answer order, formatting, or contextual cues. A judge may also favor fluent or lengthy answers over correct ones. Separate correctness from style, specify brevity where it matters, and test comparisons in both orders rather than assuming a single judgment is neutral. section 5.4.4
Table 5.2 Commonly observed biases in LLM-as-a-judge implementations, along with typical mitigation strategies.
|
Bias
|
Description
|
Mitigation
|
|
Position-bias
|
The judge
favors outputs appearing earlier (or later) in a list or comparison, regardless
of quality
|
Randomize
output order; evaluate both orders; aggregate over multiple runs.
|
|
Length bias
|
Longer
answers are often rated as better, more thorough, or more helpful, even when
they contain redundant or incorrect information.
|
Normalize
or cap length; include explicit brevity criteria.
|
|
Fluency bias
|
Well-written
text is judged better than awkward but correct text.
|
Separate
fluency from correctness in the rubric; evaluate content before style.
|
|
Self-preference
bias
|
Judge
favors outputs stylistically similar to itself.
|
Use judges
different from evaluated models.
|
|
Anchoring
bias
|
Early
examples or scores influence later judgments.
|
Reset
context frequently; limit batch size.
|
Compare common judge biases with practical mitigation strategies. section 5.4.4
Revisit the evaluator as user behavior, system use, or the judge itself changes. Prompt edits and model updates can alter judgments even when the metric's definition stays the same. Version changes to the metric and associate results with the version used; otherwise, historical scores may no longer be comparable. section 3.5




















