Completion rate measures the proportion of attempted tasks an automated system completes successfully. It is most useful when success means fulfilling a user request across multiple actions, rather than merely producing a plausible response. section 1.3.4 section 1.3.5
Interpret completion rate alongside the conditions that shape success:
- Segment it by task type to reveal whether an agent succeeds on some workflows but fails on others. section 3.4.5
- Inspect failure conditions such as timed-out tools, partial API data, long workflows, or changed user requirements. section 3.4.5
- Pair it with tool-call accuracy, instruction-following accuracy, recovery rate, or escalation rate instead of treating it as sufficient evidence alone. section 1.3.4 section 3.4.5
Conversational movie search
A streaming assistant receives a request for a movie under 90 minutes and nothing scary. Completion rate asks whether the agent actually finds a suitable title, respects both constraints, and finishes the flow. Tool-call accuracy can then show whether retrieval or ranking failures caused unsuccessful attempts. section 1.3.5
Table 1.1 Offline Metric Categories
|
Metric Category
|
Description
|
Example application
|
Common Metrics
|
|
Classification Metrics
|
To evaluate the performance of models that categorize data into predefined classes.
|
Email spam detection system
|
Accuracy, Precision, Recall, F1 Score
|
|
Ranking and Retrieval Metrics
|
To evaluate models that retrieve, sort, or rank items based on relevance to a user or query.
|
Recommendation systems, search, RAG retrieval
|
NDCG, MAP, Precision@K
|
|
Clustering Metrics
|
To evaluate models that groups data points based on similarities, without predefined labels.
|
Customer segmentation for personalization
|
Silhouette Score, Davies-Bouldin Index
|
|
Agentic System Metrics
|
To evaluate systems that plan, call tools, take actions, or complete multi-step tasks.
|
Support agent assistant, coding agent, workflow automation agent
|
Task completion rate, tool-call accuracy, step success rate, constraint adherence, human override rate
|
|
Natural Language Processing (NLP) Metrics
|
To evaluate models that process, classify, translate, transcribe, summarize, or compare human language.
|
Machine translation, speech-to-text, text classification, summarization
|
BLEU, ROUGE, METEOR, BERTScore
|
|
Generative AI / LLM Metrics
|
To evaluate generated outputs for usefulness, correctness, consistency, and alignment with user intent.
|
Customer support chatbot, summarization assistant, personalized recommendation explanations
|
Relevance, factuality, groundedness, instruction following, toxicity; often scored through human review, rubric-based evaluation, or LLM-as-a-judge
|
|
Time Series and Forecasting Metrics
|
To evaluate models that predict future values or trends based on historical data.
|
Forecasting for sales or inventory management
|
MAE, RMSE, MAPE
|
|
Image and Computer Vision Metrics
|
To evaluate models that analyze visual data, such as images or videos.
|
Product image classification on a website
|
IoU, mAP, Accuracy
|
|
Regression Metrics
|
To evaluate models that predict continuous outcomes.
|
Predicting housing prices
|
RMSE, MAE
|
Use the offline metric categories table to compare completion rate with metrics for classification, ranking, retrieval, and other agentic-system behaviors. section 1.3.4


