liveBook · Manning

LLM-as-a-judge

As explained in Vibe Engineering

LLM-as-a-Judge is an evaluation approach in which a large language model assesses or grades an artifact, change, or another model's output against specified criteria.

A model can confidently approve an error it helped create. LLM-as-a-judge uses a language model to assess an artifact or another model’s output against a prompt or specified criteria. The judgment can widen review coverage, but it is not necessarily an independent check: generator and judge may share blind spots. section 3.6.3 section 5.3.2

The quality of the specification matters more than which model grades the work. A vague requirement can produce a confident justification for the wrong behavior. Explicit, machine-checkable requirements and external feedback provide signals that do not depend solely on the model’s judgment. section 3.6.3 section 8.2.3

Useful review tasks have explicit scope and context:

  • In CI, review a pull-request diff alongside test results and dependency context. Check conventions, test coverage relative to risk, security concerns, and the significance of the changes. section 5.3.2
  • Split review among narrowly scoped security, performance, and API compatibility agents, with a team responsible for each one. Their findings supplement rather than replace human review. section 5.3.2
  • Compare a diff with the original prompt to flag out-of-scope changes, such as disabling a flaky test. A veto can catch shortcuts, but false positives and brittleness are costs. section 10.4.2
  • Before installing dependencies, prompt an agent to check registry existence, maintenance history, and suspiciously similar package names. Do not treat a confident model answer alone as proof of provenance. section 5.2.4 section 3.6.3
Listing 5.4 Pre-commit hook example
#!/bin/bash
# .git/hooks/pre-commit
set -euo pipefail
 
DIFF=$(git diff --cached)
[ -z "$DIFF" ] && exit 0
 
# Deterministic checks block. This is the real gate.
gitleaks protect --staged --redact || exit 1
 
# The model advises. It cannot block on its own.
VERDICT=$(printf '%s' "$DIFF" | claude -p "Review this diff against our
conventions: error handling, test coverage for new public functions,
naming. Reply with exactly one line of JSON and nothing else:
{\"verdict\":\"clean\"} or {\"verdict\":\"concerns\",\"notes\":\"...\"}")
 
if [ "$(printf '%s' "$VERDICT" | jq -r '.verdict')" = "concerns" ]; then
    echo "AI reviewer raised concerns (advisory, not blocking):"
    printf '%s' "$VERDICT" | jq -r '.notes'
fi
exit 0

This pre-commit hook separates the roles: a deterministic secret check blocks the commit, while the model reviews the staged diff and reports concerns without blocking. section 5.3.2

A judge with an external signal

In a study of generated COBOL across 100 programs, four judge systems alone caught 45–63% of the errors present, with consistent misses in domain-specific issues. Explicit hints from a separate non-LLM analytic checker raised coverage to 74%. The result illustrates why an outside signal can matter more than another model-only review. section 8.2.3

Keep meaningful tests, automated checks, and human review in the feedback loop. A judge can identify concerns and summarize changes, but it cannot infer requirements or architectural intent that it has not been given. section 3.6.3 section 8.2.3 section 5.3.2

How Vibe Engineering covers it

How deep it goes
Explains judge limitations and review workflows, including advisory hooks, scoped agents, and external verification signals.
Examples
Illustrates advisory pre-commit review, dependency validation, and error detection in generated COBOL.
What you can do after
Operate advisory Git-hook or CI reviewers with explicit criteria while retaining deterministic checks and human review.
How it connects
Connects self-correction, code review, continuous integration, specifications, and automated testing.

Also explained in

Compare how 21 books explain LLM-as-a-judge →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.