A model can confidently approve an error it helped create. LLM-as-a-judge uses a language model to assess an artifact or another model’s output against a prompt or specified criteria. The judgment can widen review coverage, but it is not necessarily an independent check: generator and judge may share blind spots. section 3.6.3 section 5.3.2
The quality of the specification matters more than which model grades the work. A vague requirement can produce a confident justification for the wrong behavior. Explicit, machine-checkable requirements and external feedback provide signals that do not depend solely on the model’s judgment. section 3.6.3 section 8.2.3
Useful review tasks have explicit scope and context:
- In CI, review a pull-request diff alongside test results and dependency context. Check conventions, test coverage relative to risk, security concerns, and the significance of the changes. section 5.3.2
- Split review among narrowly scoped security, performance, and API compatibility agents, with a team responsible for each one. Their findings supplement rather than replace human review. section 5.3.2
- Compare a diff with the original prompt to flag out-of-scope changes, such as disabling a flaky test. A veto can catch shortcuts, but false positives and brittleness are costs. section 10.4.2
- Before installing dependencies, prompt an agent to check registry existence, maintenance history, and suspiciously similar package names. Do not treat a confident model answer alone as proof of provenance. section 5.2.4 section 3.6.3
Listing 5.4 Pre-commit hook example
#!/bin/bash
# .git/hooks/pre-commit
set -euo pipefail
DIFF=$(git diff --cached)
[ -z "$DIFF" ] && exit 0
# Deterministic checks block. This is the real gate.
gitleaks protect --staged --redact || exit 1
# The model advises. It cannot block on its own.
VERDICT=$(printf '%s' "$DIFF" | claude -p "Review this diff against our
conventions: error handling, test coverage for new public functions,
naming. Reply with exactly one line of JSON and nothing else:
{\"verdict\":\"clean\"} or {\"verdict\":\"concerns\",\"notes\":\"...\"}")
if [ "$(printf '%s' "$VERDICT" | jq -r '.verdict')" = "concerns" ]; then
echo "AI reviewer raised concerns (advisory, not blocking):"
printf '%s' "$VERDICT" | jq -r '.notes'
fi
exit 0This pre-commit hook separates the roles: a deterministic secret check blocks the commit, while the model reviews the staged diff and reports concerns without blocking. section 5.3.2
A judge with an external signal
In a study of generated COBOL across 100 programs, four judge systems alone caught 45–63% of the errors present, with consistent misses in domain-specific issues. Explicit hints from a separate non-LLM analytic checker raised coverage to 74%. The result illustrates why an outside signal can matter more than another model-only review. section 8.2.3
Keep meaningful tests, automated checks, and human review in the feedback loop. A judge can identify concerns and summarize changes, but it cannot infer requirements or architectural intent that it has not been given. section 3.6.3 section 8.2.3 section 5.3.2




















