liveBook · Manning

LLM-as-a-judge

As explained in LLM Customization and Fine-Tuning

An evaluation setup in which a language model judges other model outputs against specified criteria.

A response can be useful while sharing few words with a reference answer—or match the reference closely while adding a false instruction. AI-as-a-judge uses a language model and an explicit rubric to assess outputs on criteria that exact matching or token overlap can miss, such as correctness, completeness, tone, and reasoning quality. section 7.2.3 section 4.4.4

A practical evaluation workflow separates inexpensive checks from judgment-intensive review:

  • Define a rubric for the behavior that matters and apply it to a fixed test set. For distillation, provide the prompt, teacher response, and student response; score correctness, completeness, and coherence from 1 to 5. section 4.4.4 section 7.2.3
  • Use automated metrics at every checkpoint to catch gross regressions, then run the judge on the final candidate before deployment. Compare results by category rather than relying only on an overall score. section 7.2.3 section 5.11
  • Choose a stronger judge than the model being evaluated, and validate a final prompt with a second judge to help expose judge-specific bias. A frontier API can serve as an offline judge even when it is too costly for every production request. section 4.4.4

Distillation: overlap versus usefulness

In an IT support evaluation, a useful paraphrase of a teacher’s VPN setup instructions scores 0.31 on Token-F1, while a vocabulary-matching answer that invents a network-service restart scores 0.83. A judge can assess whether each student response is correct and complete. High Token-F1 with a low judge score suggests surface matching; low Token-F1 with a high judge score suggests successful paraphrasing. Running a frontier judge on the same 50-example test split is estimated at $1–$3 once per distillation experiment. section 7.2.3

Judging also supports data quality. A different model can review teacher-generated outputs for correctness and completeness, while a synthetic-data quality gate can assess fidelity, format, and domain safety. In weak supervision, a judge given the annotators’ disambiguation rules and examples can contribute a labeling vote; unresolved cases go to humans. section 7.1.3 section 3.9 section 3.4.2

A judge is not a substitute for the rest of an evaluation system. Its biases affect its scores, and cross-model review should supplement human spot-checking on the initial iteration. Combine qualitative judgments with deterministic checks, quantitative metrics, safety tests, and human review when the decision warrants them. section 4.4.4 section 7.1.3 section 6.3.2 section 5.11

How LLM Customization and Fine-Tuning covers it

How deep it goes
Specifies judge inputs, scoring dimensions, test-set comparison, model selection, and evaluation cadence without providing a judge implementation.
Examples
Compares VPN instruction paraphrases and invented steps, then scores student responses for correctness, completeness, and coherence.
What you can do after
Evaluate model outputs using a rubric, compare judge scores with Token-F1 by category, and identify possible surface matching or successful paraphrasing.
How it connects
Connects rubric-based evaluation with knowledge distillation, Token-F1, weak supervision, and human review.

Also explained in

Compare how 21 books explain LLM-as-a-judge →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.