liveBook · Manning

Scoring mechanism

As explained in AI Model Evaluation

An evaluation technique that assigns numerical values to one or more dimensions of a model output.

A scoring mechanism assigns numerical values to one or more dimensions of a model output. It turns qualities such as factuality, helpfulness, safety, or task success into signals that can be aggregated and compared. The useful score is the one aligned with the product decision: for example, whether a recommendation is relevant or whether an agent completes its task reliably. section 1.3.4

LLM-as-a-judge is a scoring method, not a metric category. A judge can score groundedness, relevance, factual consistency, instruction following, or task success using a rubric or grading prompt. Treat its output as an evaluation signal rather than absolute truth: validate and calibrate it before using it to guide model or product decisions. section 1.3.4

Choose the output format according to the decision you need:

  • Use scoring, such as 1–5 values, to track aggregate trends across many examples, including average factuality, helpfulness, or safety. section 9.5.3
  • Use ranking to order several candidates for the same input, such as recommendations, summaries, or agent plans; account for presentation order, length, and style effects. section 9.5.3
  • Use pairwise comparison to decide whether one response, model variant, or prompt beats another; expect higher judge-call costs at scale. section 9.5.3

Recommendation evaluation

For a recommendation system, score each output for relevance and aggregate the results across users to monitor trends. Compare two model variants pairwise when choosing a new baseline, then inspect individual failures rather than treating the average as complete evidence of user value. section 1.3.4 section 9.5.3

A number can hide the reason for failure. A response with one dangerous hallucination may receive a similar score to a response that is broadly good but error-free. Combine checklist items for concrete requirements such as factuality or policy compliance with scores for holistic qualities such as tone or helpfulness. section 10.2.1

Table 1.1 Offline Metric Categories
Metric Category
Description
Example application
Common Metrics
Classification Metrics
To evaluate the performance of models that categorize data into predefined classes.
Email spam detection system
Accuracy, Precision, Recall, F1 Score
Ranking and Retrieval Metrics
To evaluate models that retrieve, sort, or rank items based on relevance to a user or query.
Recommendation systems, search, RAG retrieval
NDCG, MAP, Precision@K
Clustering Metrics
To evaluate models that groups data points based on similarities, without predefined labels.
Customer segmentation for personalization
Silhouette Score, Davies-Bouldin Index
Agentic System Metrics
To evaluate systems that plan, call tools, take actions, or complete multi-step tasks.
Support agent assistant, coding agent, workflow automation agent
Task completion rate, tool-call accuracy, step success rate, constraint adherence, human override rate
Natural Language Processing (NLP) Metrics
To evaluate models that process, classify, translate, transcribe, summarize, or compare human language.
Machine translation, speech-to-text, text classification, summarization
BLEU, ROUGE, METEOR, BERTScore
Generative AI / LLM Metrics
To evaluate generated outputs for usefulness, correctness, consistency, and alignment with user intent.
Customer support chatbot, summarization assistant, personalized recommendation explanations
Relevance, factuality, groundedness, instruction following, toxicity; often scored through human review, rubric-based evaluation, or LLM-as-a-judge
Time Series and Forecasting Metrics
To evaluate models that predict future values or trends based on historical data.
Forecasting for sales or inventory management
MAE, RMSE, MAPE
Image and Computer Vision Metrics
To evaluate models that analyze visual data, such as images or videos.
Product image classification on a website
IoU, mAP, Accuracy
Regression Metrics
To evaluate models that predict continuous outcomes.
Predicting housing prices
RMSE, MAE

Use the metric-category table to match the scoring target to the system being evaluated, from ranking and retrieval to agentic task completion. section 1.3.4

How AI Model Evaluation covers it

How deep it goes
Explains scoring, ranking, pairwise comparison, calibration, and checklist trade-offs, but stops short of formal calibration procedures.
Examples
Uses recommendation relevance, factuality, helpfulness, safety, model variants, summaries, and agent plans as evaluation examples.
What it assumes
Assumes a defined quality dimension and product goal, while distinguishing those targets from the method used to score them.
What you can do after
Evaluate model outputs with intentional scoring, ranking, pairwise comparison, and checklist formats.
How it connects
Connects metrics with LLM-as-a-judge, ranking, pairwise comparison, checklists, calibration, and product evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.