liveBook · Manning

Loop/retry rate

As explained in AI Model Evaluation

The frequency with which an agent system repeats an execution loop during task completion.

Loop/retry rate measures how often an agent repeats execution steps or re-attempts failed actions during one task. It reveals repeated work that can make a system feel slow or unreliable, even when individual model calls are fast. section 4.2.3

Interpret the metric alongside related agent-performance measures:

  • A high rate can indicate brittle tool integrations or unclear task instructions. section 4.2.3
  • Pair it with model-call count, tool-call count, agent step count, tool failure rate, external dependency failures, and total task completion time. section 4.2.3 section 4.6
  • Use it to locate where cumulative task time and unreliability originate rather than judging model speed alone. section 4.2.3

Reading a high rate

If an agent repeatedly retries a failed external action, the loop/retry rate highlights a likely integration or instruction problem that merits investigation. section 4.2.3

How AI Model Evaluation covers it

How deep it goes
Defines repeated steps and failed-action retries, then links high rates to brittle integrations or unclear instructions.
Examples
Uses repeated steps and re-attempted failed actions as the measured events.
What it assumes
Assumes familiarity with agent tasks, model calls, tool calls, and execution steps.
What you can do after
Diagnose/debug with it by locating repeated actions that may signal brittle tool integrations or unclear task instructions.
How it connects
Connects agent step count, tool failure rate, external dependency failures, latency, and total task completion time.

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.