liveBook · Manning

Unit of evaluation

As explained in AI Model Evaluation

A unit of evaluation is the individual entity at which a model or system is assessed, such as a query, session, response, or completed task.

A unit of evaluation is the entity at which a model or system is assessed. Choosing it makes an offline evaluation concrete, repeatable, and aligned with the product question: assess a user, query, session, recommendation list, generated response, conversation, or completed task. section 2.2.2

Choose the unit to match what the system is meant to improve:

  • Use a recommendation list when evaluating ranking quality with measures such as Precision@K or NDCG. section 2.2.2
  • Use a generated response when assessing groundedness, relevance, factuality, or instruction following. section 2.2.2
  • Use a completed task when measuring an agent’s task completion, tool-call accuracy, constraint adherence, or escalation quality. section 2.2.2

Recommendation evaluation

To evaluate a recommendation system, assess recommendation lists rather than treating all activity as one undifferentiated result. Include data-sparse cases such as cold-start users so the unit reflects important real-world conditions. section 2.2.2

How AI Model Evaluation covers it

How deep it goes
Defines evaluation granularity and connects it to objectives, metrics, scenarios, baselines, and decision thresholds.
Examples
Uses users, queries, sessions, recommendation lists, generated responses, conversations, completed tasks, and cold-start scenarios.
What it assumes
Assumes a defined evaluation objective and a comparison baseline before selecting the unit.
What you can do after
Apply it via offline evaluation design by selecting an entity, matching metrics to objectives, and testing relevant scenarios.
How it connects
Connects offline evaluation with metrics, baselines, scenarios, fairness, robustness, groundedness, and task completion.

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.