An LLM-as-a-Judge uses a powerful language model to assess generated outputs against criteria. In a retrieval-augmented generation (RAG) app, it can help scale end-to-end evaluation, complementing component-level tests. section 7.3.2
For a generated response, assess:
- Groundedness: whether claims rely on the retrieved documents rather than speculation or hallucination. section 7.3.2
- Answer relevance: whether the response meets the query’s requested focus and depth. section 7.3.2
- Instruction adherence: whether a fine-tuned model follows the given prompts. section 8.2.3
Evaluating a generated report
For a report on AI trends in SaaS, a judge checks whether the response is supported by retrieved sources and whether it gives the depth the user requested. For instruction fine-tuning, a judge can score adherence on a held-out 20% test set. section 7.3.2 section 8.2.3
Treat judge scores as one part of evaluation: manual audits and user feedback also help identify recurring issues. Keep criteria responsive to real use, and add emerging concerns such as style or tone with oversight. section 7.3.2 section 8.2.3
