liveBook · Manning

LLM-as-a-judge

As explained in AI Governance

A language model used to evaluate another model’s plan or output against specified criteria.

An LLM-as-a-judge uses a second model to assess another model’s output or plan against a reference or approved goal. It can trigger regeneration or human review when it detects a problem, but adds inference cost and can make incorrect or biased judgments. section 6.1.1 section 6.7.1

Use the judge as one control in a risk-aware workflow:

  • For consequential outputs, compare candidate judges with human-labeled decisions and tune the chosen judge’s prompt. section 6.1.1
  • For live agent workflows, check plans or outputs against the approved version and escalate deviations; batch evaluation tools may not keep pace. section 6.7.1
  • Treat judge results as fallible; consider multiple judges for critical decisions without assuming they are independent. section 6.7.1

ClaimAssist

ElectraMotors evaluated several judges against human-labeled data in MLFlow, then tuned the selected judge’s prompt to better match human decisions. section 6.1.1

How AI Governance covers it

How deep it goes
Explains judge validation, prompt tuning, cost, and bias, but does not specify a judge architecture.
Examples
Uses ClaimAssist judge comparisons against human-labeled data and checks agent plans against approved versions.
What you can do after
Evaluate judges against human decisions, tune prompts, and route flagged outputs or plans to review.
How it connects
Connects model evaluation with hallucination detection, human oversight, regeneration, and agent plan approval.

Also explained in

Compare how 21 books explain LLM-as-a-judge →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.