liveBook · Manning

Prompt

As explained in AI Model Evaluation

A prompt is text or other input provided to a generative model to specify a task or elicit a response.

A prompt is input that tells a generative model what task to perform. In an LLM-as-a-judge evaluation, the prompt acts like an evaluation form: it identifies what to inspect, which criteria to apply, and how to return a judgment. The prompt and its context shape what the judge can consider and what counts as a good result. section 9.4 section 9.5.2

Before writing a judge prompt, define the decision it should inform, the data it can compare, and the output needed for analysis. A specific goal still needs representative data; a vague goal can produce noisy judgments even with strong data. section 9.4

A judge prompt typically has four parts:

  • Context: provide the information needed to judge, such as the user query, source document, model response, or candidate recommendations. section 9.5.2
  • Role and instruction: state what the judge is doing and what it should evaluate. section 9.5.2
  • Rubric: define the criteria and scoring rules rather than leaving terms such as factuality, helpfulness, or safety open to interpretation. section 9.5.2
  • Output schema: specify the result format, such as JSON fields for a score, label, winner, and justification. section 9.5.2

Judging movie recommendations

A prompt that asks a judge to assess how good recommendations are, given a user's viewing history, leaves the criteria and format unspecified. One response might call the recommendations solid and rate them either 7 or 8 out of 10. For a top-5 evaluation, specify relevance and diversity criteria and a consistent output format so results can be compared across models. section 9.6.1 section 9.5.2

Avoid collapsing several dimensions into a single preference too early. Score dimensions separately before requesting an overall decision. For a scalable pipeline, use a machine-readable format that can be parsed and aggregated. Start with a baseline, test it on examples with expected judgments, inspect disagreements, and change one part at a time. Version prompt changes and document why they were made. section 9.5.2

Figure 9.1 The three essential components of an evaluation system are goal, data, and evaluator. They work together to determine how reliable and actionable your results are. Each term in this equation carries equal weight. The goal defines what decision the evaluation is meant to inform. The data determines the evidence available to answer that question, from offline samples to live production logs. The evaluator, whether human, metric-based, or an LLM-as-a-judge, applies criteria and interpretation to that evidence. Getting the goal and data right upfront keeps the rest of the design process (prompting, calibration, engineering) on solid ground.

See how the evaluation goal, data, and evaluator work together; a prompt alone does not make an evaluation reliable. section 9.4

How AI Model Evaluation covers it

How deep it goes
Details prompt components and iteration steps, including rubrics, structured output, baseline testing, and versioning.
Examples
Contrasts an underspecified movie-recommendation judging prompt with the need for explicit relevance, diversity, and output criteria.
What you can do after
Apply prompt design to an LLM-as-a-judge evaluation by specifying context, criteria, and output, then testing and revising the baseline.
How it connects
Connects prompts to LLM-as-a-judge evaluation, rubrics, reference grounding, and structured output.

Also explained in

Compare how 64 books explain prompt →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.