liveBook · Manning

LLM-as-a-judge

As explained in Designing AI Agents

Model-based judging is the assessment of outputs using a model as the judge.

LLM-as-a-judge uses a separate model to score an output. Treat its score as one quality measurement, not ground truth. section 9.5.3

Judge scores need calibration and safeguards:

  • Compare scores with human evaluations; judges can favor outputs from their own model family. section 9.5.3
  • Record criterion-level findings rather than relying on one unexplained score; verify findings against evidence. section 7.2.4

How Designing AI Agents covers it

How deep it goes
Describes separate-model scoring and calibration needs without specifying a scoring procedure.
What you can do after
Evaluate outputs with a separate model while treating its score as one measurement rather than ground truth.
How it connects
Connects model-based judging to human evaluation, calibration, and self-preference bias.

Also explained in

Compare how 21 books explain LLM-as-a-judge →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.