Evaluation flow

As explained in AI Agents in Action

An evaluation flow is a workflow that computes or aggregates assessments of prompts, outputs, or use cases.

An evaluation flow is a workflow that runs after a generative flow to score outputs against criteria and aggregate the results. In prompt flow, it uses Python blocks rather than LLM blocks for scoring and aggregation. section 9.7.3 section 9.8

A practical evaluation flow typically:

• Processes each result, converts numeric criterion scores, and calculates an average score. section 9.7.3
• Aggregates criterion values across results and logs summary metrics. section 9.7.3
• Runs against an existing batch run so prompt or profile variations can be compared. section 9.7.3

Comparing recommendation prompts

Run recommendation variations in batch mode, select a previous run for evaluation, execute the evaluation flow, and compare aggregated criteria in Visualize Runs. A separate flow can also evaluate multiple generated plans and select the most consistent answer. section 9.7.3 section 10.3.1

Figure 9.24 Looking at the evaluate_groundings flow used to ground recommendation runs

Inspect the flow diagram to see how Python scoring and aggregation blocks form a reusable evaluation workflow. section 9.7.3

How AI Agents in Action covers it

How deep it goes

Explains scoring, aggregation, batch execution, and run comparison, but defers detailed rubric design and prompt tactics.

Examples

Uses recommendation runs, prompt variants, custom rubric criteria, and repeated plans as evaluation examples.

What it assumes

Assumes familiarity with prompt flow batch runs while teaching evaluation through scoring, aggregation, and practical exercises.

What you can do after

Operate evaluation flows via prompt flow, score batch outputs, aggregate criteria, and compare runs.

How it connects

Connects prompt flow with rubrics, grounding, batch processing, DAGs, and self-consistency prompting.

liveBookmanning.com

Read AI Agents in Action on liveBook Get the book at manning.com

Related concepts

• self-consistencyuses · Self-consistency is a technique that generates multiple reasoning paths or responses for the same input and selects or combines results based on their agreement.
• batch sizeuses · Batch size is the number of items grouped together for processing or computation as a single batch.

This explainer was generated by Manning's knowledge graph from the text of AI Agents in Action. The section links point to the parts of the book it draws on.