Tool-use correctness measures whether an agent selects and invokes the appropriate tool for the user’s request. It belongs to agentic-system evaluation alongside task completion rate and step success rate, rather than serving as a universal quality measure. section 1.3.4
Treat it as a product-facing offline metric: choose it when the experience depends on an agent calling the right retrieval or ranking tool. Interpret it with the broader task outcome, because a correct tool call alone does not establish that the user’s request was completed successfully. section 1.3.5
Movie-finding assistant
A conversational movie assistant receives a request such as finding something that is not scary and under 90 minutes. Tool-use correctness asks whether it calls the appropriate retrieval or ranking tool while respecting the request’s constraints; task completion rate asks whether the user actually finds a matching movie. section 1.3.5
Table 1.1 Offline Metric Categories
|
Metric Category
|
Description
|
Example application
|
Common Metrics
|
|
Classification Metrics
|
To evaluate the performance of models that categorize data into predefined classes.
|
Email spam detection system
|
Accuracy, Precision, Recall, F1 Score
|
|
Ranking and Retrieval Metrics
|
To evaluate models that retrieve, sort, or rank items based on relevance to a user or query.
|
Recommendation systems, search, RAG retrieval
|
NDCG, MAP, Precision@K
|
|
Clustering Metrics
|
To evaluate models that groups data points based on similarities, without predefined labels.
|
Customer segmentation for personalization
|
Silhouette Score, Davies-Bouldin Index
|
|
Agentic System Metrics
|
To evaluate systems that plan, call tools, take actions, or complete multi-step tasks.
|
Support agent assistant, coding agent, workflow automation agent
|
Task completion rate, tool-call accuracy, step success rate, constraint adherence, human override rate
|
|
Natural Language Processing (NLP) Metrics
|
To evaluate models that process, classify, translate, transcribe, summarize, or compare human language.
|
Machine translation, speech-to-text, text classification, summarization
|
BLEU, ROUGE, METEOR, BERTScore
|
|
Generative AI / LLM Metrics
|
To evaluate generated outputs for usefulness, correctness, consistency, and alignment with user intent.
|
Customer support chatbot, summarization assistant, personalized recommendation explanations
|
Relevance, factuality, groundedness, instruction following, toxicity; often scored through human review, rubric-based evaluation, or LLM-as-a-judge
|
|
Time Series and Forecasting Metrics
|
To evaluate models that predict future values or trends based on historical data.
|
Forecasting for sales or inventory management
|
MAE, RMSE, MAPE
|
|
Image and Computer Vision Metrics
|
To evaluate models that analyze visual data, such as images or videos.
|
Product image classification on a website
|
IoU, mAP, Accuracy
|
|
Regression Metrics
|
To evaluate models that predict continuous outcomes.
|
Predicting housing prices
|
RMSE, MAE
|
Use the offline metric categories table to place tool-use correctness among measures for systems that plan, call tools, take actions, or complete multi-step tasks. section 1.3.4
