A unit of evaluation is the entity at which a model or system is assessed. Choosing it makes an offline evaluation concrete, repeatable, and aligned with the product question: assess a user, query, session, recommendation list, generated response, conversation, or completed task. section 2.2.2
Choose the unit to match what the system is meant to improve:
- Use a recommendation list when evaluating ranking quality with measures such as Precision@K or NDCG. section 2.2.2
- Use a generated response when assessing groundedness, relevance, factuality, or instruction following. section 2.2.2
- Use a completed task when measuring an agent’s task completion, tool-call accuracy, constraint adherence, or escalation quality. section 2.2.2
Recommendation evaluation
To evaluate a recommendation system, assess recommendation lists rather than treating all activity as one undifferentiated result. Include data-sparse cases such as cold-start users so the unit reflects important real-world conditions. section 2.2.2
