A rubric is a structured set of criteria and standards for specifying or evaluating how well an agent profile or prompt meets its objectives. It replaces a simple right-or-wrong measure with criteria suited to the use case. section 9.5
Build a practical rubric by following these steps:
- Identify the profile or agent’s purpose and objectives. section 9.5
- Define specific, measurable criteria aligned with those objectives. section 9.5
- Create a numerical or descriptive performance scale. section 9.5
- Describe what strong and weak performance looks like at every level. section 9.5
- Apply the rubric, assign criterion scores, and calculate a sum or weighted average when appropriate. section 9.5
Evaluating recommendations
A recommendation rubric can score genre, subject, and format from 1 to 5. A score of 1 means poor alignment, while 5 means excellent alignment. A second LLM prompt can evaluate each generated recommendation automatically, producing scores such as Subject: 5, Format: 5, and Genre: 4. section 9.5 section 9.6
Table 9.2 Rubric ratings
|
Rating
|
Description
|
|---|---|
|
1
|
Poor alignment: this is the opposite of what is expected given the criteria.
|
|
2
|
Bad alignment: this isn’t a good fit for the given criteria.
|
|
3
|
Mediocre alignment: it may or may not fit well with the given criteria.
|
|
4
|
Good alignment: it may not align 100% with the criteria but is a good fit otherwise.
|
|
5
|
Excellent alignment: this is a good recommendation for the given criteria.
|
Use the rating table to compare alignment levels consistently across evaluated recommendations. section 9.5
In Prompt flow, a rubric can become an evaluation prompt. An evaluation flow runs after a generative flow, scores results, and aggregates criteria across single or batch runs. section 9.6 section 9.8
