liveBook · Manning

Discovery

As explained in AI Model Evaluation

A measure of whether recommendations introduce items that are new or unexpected while remaining relevant to the user.

Discovery measures whether recommendations introduce items that feel new or unexpected while remaining relevant. It complements relevance and diversity by focusing on useful novelty rather than simple variety. section 9.6.1

Evaluate discovery in a structured recommendation-quality check:

  • Provide the user's viewing history and the model's top-five recommendations. section 9.6.1
  • Ask for a one- to two-sentence justification and a score from 1 to 5. section 9.6.1
  • Calibrate judgments with human- or expert-scored examples, then compare rerun stability, inter-model agreement, or correlation with human ratings. section 9.6.1

How AI Model Evaluation covers it

How deep it goes
Defines a 1–5 judged dimension and adds calibration and reproducibility checks, but does not specify an algorithmic discovery formula.
Examples
Uses top-five movie recommendations and highlights introducing lesser-known directors as a high-discovery outcome.
What you can do after
Evaluate recommendation novelty with structured scores, concise justifications, calibrated examples, and machine-readable JSON.
How it connects
Connects discovery to relevance, diversity, LLM-as-a-judge, prompt calibration, and human ratings.

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.