Evaluation checks how well an agent, prompt, solution, action, or outcome performs against a rubric, expected answer, or task goal. It can annotate reasoning, guide planning, and determine whether a task is complete. section 1.2 section 1.3
Practical evaluation approaches include:
- Compare prompt or profile variations in parallel to assess how changes to templates, models, temperature, or token limits affect performance. section 9.4
- Score generated recommendations with a rubric, either manually for a baseline or automatically through a second LLM prompt. section 9.6
- Measure question-answering quality by comparing embeddings of predicted and expected answers and producing a similarity score. section 10.1.1
- Use external monitoring, internal feedback, or manual review to evaluate personal assistants, service bots, autonomous agents, collaborative workflows, and game AI. section 11.5.3
Evaluating a recommendation profile
A recommendation flow generates movie suggestions, then passes each suggestion to an evaluation prompt. The rubric scores subject, format, and genre, producing results such as 5, 5, and 4. The same rubric can compare profile variations and establish a repeatable baseline. section 9.6 section 9.7
Table 11.3 When and where evaluation is employed and used in various applications
|
Application
|
Implemented
|
Environment
|
Purpose
|
Timing
|
Configuration
|
|---|---|---|---|---|---|
|
Personal assistant
|
External
|
Server
|
Determine how well the system is working
|
After the interaction
|
Often developed externally
|
|
Customer service bot
|
External monitor
|
Server
|
Evaluate the success of each interaction
|
After the interaction
|
External to the agent system
|
|
Autonomous agent
|
External or internal
|
Server or service
|
Determine the success of the system after or during task completion
|
After the interaction
|
External or internal
|
|
Collaborative workflows
|
External
|
Shared canvas or coding
|
Evaluate the success of the collaboration
|
After the interaction
|
External service
|
|
Game AI
|
External or internal
|
Server or application
|
Evaluate the agent or evaluate the success of a strategy or action
|
After the interaction
|
External or as part of the agent or another agent
|
|
Research
|
Combined manual and LLM
|
Server and human
|
Evaluate the output of the research developed
|
After the generated output
|
Depends on the complexity of the problem and research undertaken
|
Use the application matrix to compare where evaluation runs, when it runs, and whether it is external or internal. section 11.5.3
