Prompt engineering deliberately designs and refines model instructions to produce consistent behavior. For LLM-as-a-judge systems, effective prompts define the task, scoring guidance, examples, and explicit quality rubrics; vague instructions produce unreliable evaluations. section 4.2.1
Build a prompt around independently adjustable evaluation components:
- Define the task and what quality means in the specific context. section 4.2.1
- Specify detailed criteria or a rubric rather than relying on vague instructions. section 4.2.5
- Add illustrative examples with correct answers to make judgments more consistent. section 4.2.1
- Refine task definitions, criteria, and reasoning steps against domain-expert preferences. section 4.2.5
Reducing hallucinations with instructions
A prompt can instruct a model to use only information from supplied documents and to say when it does not know. These instructions reduce hallucinations cheaply, but models may ignore them when prompts become long or complex. section 5.4.5
Figure 4.2 Detailed instructions (top) and examples (bottom) provided to human labelers at Amazon Mechanical Turk. This approach of providing both explicit guidelines and illustrative examples is essential for high-quality human annotation and is a best practice when prompting LLM-as-a-judge. Source: Lee et al. (2024)
Compare detailed evaluator instructions with illustrative examples, two prompt components that support consistent judgments. section 4.2.1


















































