An evaluation module is the agent’s real-time monitor. After each execution step, it reads the shared workspace and checks whether results are useful, consistent, and moving toward the goal. Unlike Reflexion, which corrects a completed response, evaluation catches problems before they compound. section 10.2.6 section 10.2.1
A practical evaluation pass asks:
- Did this step make progress toward the task? section 10.2.6
- Are the new findings consistent with earlier evidence? section 10.2.6
- Should confidence increase or decrease, and should execution continue or change course? section 10.2.6
Listing 10.5 05_evaluation_module.py
class EvaluationResult(BaseModel):
progress_assessment: str #1
consistency_check: bool #2
confidence_delta: float #3
contradictions: list[str] = Field(default_factory=list)
recommendation: str #4
evaluation_agent = Agent(
name="Evaluator",
instructions="""You are the evaluation module of a cognitive agent.
After each execution step, you assess the quality of the results
and the overall trajectory of the task.
Assess:
1. progress_assessment: Did this step advance us toward the goal?
Be specific about what was gained or not gained.
2. consistency_check: Are the new results consistent with previous
findings? If not, flag contradictions.
3. confidence_delta: Should confidence go up or down based on this
step? Return a value from -0.3 to +0.3.
4. contradictions: List any conflicts between new and existing
evidence.
5. recommendation: One of CONTINUE, REPLAN, ESCALATE, TERMINATE.
- CONTINUE: Progress is good, proceed with current plan.
- REPLAN: Evidence suggests current approach is wrong.
- ESCALATE: Agent cannot resolve this, surface uncertainty.
- TERMINATE: Goal is achieved with sufficient confidence.
Be ruthless about quality. A finding with a low relevance_score
or a quality_note flagging metadata should NOT increase confidence.
Two consecutive steps with no new information should trigger
REPLAN, not CONTINUE.""",
output_type=EvaluationResult,
)
The typed evaluation result makes monitoring inspectable: it records progress, consistency, confidence change, contradictions, and the next recommendation. section 10.2.6
Testing competing causes of intermittent failures
An agent tests connection-pool exhaustion first and finds relevant official documentation. The evaluator marks progress as good, raises confidence from 0.5 to 0.65, finds no contradiction, and recommends CONTINUE. A weaker community blog about health-check races produces only a small confidence increase. After testing the hypotheses, evaluation signals TASK_COMPLETE and supports a ranked diagnosis with concrete diagnostic steps. section 10.3.3
