liveBook · Manning

Tool-use correctness

As explained in AI Model Evaluation

Tool-call accuracy measures whether an agent selects and invokes the appropriate tool correctly.

Tool-use correctness measures whether an agent selects and invokes the appropriate tool for the user’s request. It belongs to agentic-system evaluation alongside task completion rate and step success rate, rather than serving as a universal quality measure. section 1.3.4

Treat it as a product-facing offline metric: choose it when the experience depends on an agent calling the right retrieval or ranking tool. Interpret it with the broader task outcome, because a correct tool call alone does not establish that the user’s request was completed successfully. section 1.3.5

Movie-finding assistant

A conversational movie assistant receives a request such as finding something that is not scary and under 90 minutes. Tool-use correctness asks whether it calls the appropriate retrieval or ranking tool while respecting the request’s constraints; task completion rate asks whether the user actually finds a matching movie. section 1.3.5

Table 1.1 Offline Metric Categories
Metric Category
Description
Example application
Common Metrics
Classification Metrics
To evaluate the performance of models that categorize data into predefined classes.
Email spam detection system
Accuracy, Precision, Recall, F1 Score
Ranking and Retrieval Metrics
To evaluate models that retrieve, sort, or rank items based on relevance to a user or query.
Recommendation systems, search, RAG retrieval
NDCG, MAP, Precision@K
Clustering Metrics
To evaluate models that groups data points based on similarities, without predefined labels.
Customer segmentation for personalization
Silhouette Score, Davies-Bouldin Index
Agentic System Metrics
To evaluate systems that plan, call tools, take actions, or complete multi-step tasks.
Support agent assistant, coding agent, workflow automation agent
Task completion rate, tool-call accuracy, step success rate, constraint adherence, human override rate
Natural Language Processing (NLP) Metrics
To evaluate models that process, classify, translate, transcribe, summarize, or compare human language.
Machine translation, speech-to-text, text classification, summarization
BLEU, ROUGE, METEOR, BERTScore
Generative AI / LLM Metrics
To evaluate generated outputs for usefulness, correctness, consistency, and alignment with user intent.
Customer support chatbot, summarization assistant, personalized recommendation explanations
Relevance, factuality, groundedness, instruction following, toxicity; often scored through human review, rubric-based evaluation, or LLM-as-a-judge
Time Series and Forecasting Metrics
To evaluate models that predict future values or trends based on historical data.
Forecasting for sales or inventory management
MAE, RMSE, MAPE
Image and Computer Vision Metrics
To evaluate models that analyze visual data, such as images or videos.
Product image classification on a website
IoU, mAP, Accuracy
Regression Metrics
To evaluate models that predict continuous outcomes.
Predicting housing prices
RMSE, MAE

Use the offline metric categories table to place tool-use correctness among measures for systems that plan, call tools, take actions, or complete multi-step tasks. section 1.3.4

How AI Model Evaluation covers it

How deep it goes
Defines the metric and situates it among agentic measures, but does not provide a calculation, implementation procedure, or validation protocol.
Examples
Uses a conversational movie-finding request with constraints such as avoiding scary titles and staying under 90 minutes.
What you can do after
Evaluate whether an agent selects the appropriate retrieval or ranking tool for a user request.
How it connects
Connects tool-use correctness with task completion rate, constraint adherence, step success rate, retrieval, and ranking.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.