An LLM-as-a-judge uses a second model to assess another model’s output or plan against a reference or approved goal. It can trigger regeneration or human review when it detects a problem, but adds inference cost and can make incorrect or biased judgments. section 6.1.1 section 6.7.1
Use the judge as one control in a risk-aware workflow:
- For consequential outputs, compare candidate judges with human-labeled decisions and tune the chosen judge’s prompt. section 6.1.1
- For live agent workflows, check plans or outputs against the approved version and escalate deviations; batch evaluation tools may not keep pace. section 6.7.1
- Treat judge results as fallible; consider multiple judges for critical decisions without assuming they are independent. section 6.7.1
ClaimAssist
ElectraMotors evaluated several judges against human-labeled data in MLFlow, then tuned the selected judge’s prompt to better match human decisions. section 6.1.1




















