A response can be useful while sharing few words with a reference answer—or match the reference closely while adding a false instruction. AI-as-a-judge uses a language model and an explicit rubric to assess outputs on criteria that exact matching or token overlap can miss, such as correctness, completeness, tone, and reasoning quality. section 7.2.3 section 4.4.4
A practical evaluation workflow separates inexpensive checks from judgment-intensive review:
- Define a rubric for the behavior that matters and apply it to a fixed test set. For distillation, provide the prompt, teacher response, and student response; score correctness, completeness, and coherence from 1 to 5. section 4.4.4 section 7.2.3
- Use automated metrics at every checkpoint to catch gross regressions, then run the judge on the final candidate before deployment. Compare results by category rather than relying only on an overall score. section 7.2.3 section 5.11
- Choose a stronger judge than the model being evaluated, and validate a final prompt with a second judge to help expose judge-specific bias. A frontier API can serve as an offline judge even when it is too costly for every production request. section 4.4.4
Distillation: overlap versus usefulness
In an IT support evaluation, a useful paraphrase of a teacher’s VPN setup instructions scores 0.31 on Token-F1, while a vocabulary-matching answer that invents a network-service restart scores 0.83. A judge can assess whether each student response is correct and complete. High Token-F1 with a low judge score suggests surface matching; low Token-F1 with a high judge score suggests successful paraphrasing. Running a frontier judge on the same 50-example test split is estimated at $1–$3 once per distillation experiment. section 7.2.3
Judging also supports data quality. A different model can review teacher-generated outputs for correctness and completeness, while a synthetic-data quality gate can assess fidelity, format, and domain safety. In weak supervision, a judge given the annotators’ disambiguation rules and examples can contribute a labeling vote; unresolved cases go to humans. section 7.1.3 section 3.9 section 3.4.2
A judge is not a substitute for the rest of an evaluation system. Its biases affect its scores, and cross-model review should supplement human spot-checking on the initial iteration. Combine qualitative judgments with deterministic checks, quantitative metrics, safety tests, and human review when the decision warrants them. section 4.4.4 section 7.1.3 section 6.3.2 section 5.11




















