liveBook · Manning

Completion rate

As explained in AI Model Evaluation

Task completion rate is the proportion of attempted tasks that an automated system completes successfully.

Completion rate measures the proportion of attempted tasks an automated system completes successfully. It is most useful when success means fulfilling a user request across multiple actions, rather than merely producing a plausible response. section 1.3.4 section 1.3.5

Interpret completion rate alongside the conditions that shape success:

  • Segment it by task type to reveal whether an agent succeeds on some workflows but fails on others. section 3.4.5
  • Inspect failure conditions such as timed-out tools, partial API data, long workflows, or changed user requirements. section 3.4.5
  • Pair it with tool-call accuracy, instruction-following accuracy, recovery rate, or escalation rate instead of treating it as sufficient evidence alone. section 1.3.4 section 3.4.5

Conversational movie search

A streaming assistant receives a request for a movie under 90 minutes and nothing scary. Completion rate asks whether the agent actually finds a suitable title, respects both constraints, and finishes the flow. Tool-call accuracy can then show whether retrieval or ranking failures caused unsuccessful attempts. section 1.3.5

Table 1.1 Offline Metric Categories
Metric Category
Description
Example application
Common Metrics
Classification Metrics
To evaluate the performance of models that categorize data into predefined classes.
Email spam detection system
Accuracy, Precision, Recall, F1 Score
Ranking and Retrieval Metrics
To evaluate models that retrieve, sort, or rank items based on relevance to a user or query.
Recommendation systems, search, RAG retrieval
NDCG, MAP, Precision@K
Clustering Metrics
To evaluate models that groups data points based on similarities, without predefined labels.
Customer segmentation for personalization
Silhouette Score, Davies-Bouldin Index
Agentic System Metrics
To evaluate systems that plan, call tools, take actions, or complete multi-step tasks.
Support agent assistant, coding agent, workflow automation agent
Task completion rate, tool-call accuracy, step success rate, constraint adherence, human override rate
Natural Language Processing (NLP) Metrics
To evaluate models that process, classify, translate, transcribe, summarize, or compare human language.
Machine translation, speech-to-text, text classification, summarization
BLEU, ROUGE, METEOR, BERTScore
Generative AI / LLM Metrics
To evaluate generated outputs for usefulness, correctness, consistency, and alignment with user intent.
Customer support chatbot, summarization assistant, personalized recommendation explanations
Relevance, factuality, groundedness, instruction following, toxicity; often scored through human review, rubric-based evaluation, or LLM-as-a-judge
Time Series and Forecasting Metrics
To evaluate models that predict future values or trends based on historical data.
Forecasting for sales or inventory management
MAE, RMSE, MAPE
Image and Computer Vision Metrics
To evaluate models that analyze visual data, such as images or videos.
Product image classification on a website
IoU, mAP, Accuracy
Regression Metrics
To evaluate models that predict continuous outcomes.
Predicting housing prices
RMSE, MAE

Use the offline metric categories table to compare completion rate with metrics for classification, ranking, retrieval, and other agentic-system behaviors. section 1.3.4

How AI Model Evaluation covers it

How deep it goes
Defines successful completion and extends measurement into task-type and failure-mode diagnostics.
Examples
Uses conversational movie search, travel booking, account updates, and document summarization as agentic task examples.
What you can do after
Evaluate agent robustness by measuring successful workflows and diagnosing where tools, task length, data, or changing requirements cause failure.
How it connects
Connects task completion with agentic systems, tool-call accuracy, instruction following, constraint adherence, and escalation.

Also explained in

Compare how 3 books explain completion rate →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.