liveBook · Manning

Error rate

As explained in AI Model Evaluation

Error rate is the proportion of predictions or decisions that are incorrect for a defined task.

Error rate measures the proportion of incorrect predictions or decisions for a defined task. For classification, it is straightforward; recommendation systems require an explicit definition of error. section 3.2.1

Recommendation errors can represent different failures:

  • An irrelevant recommendation: a recommended item the user would not consider relevant. section 3.2.1
  • A missed relevant item: an item the user would value but the system failed to recommend. section 3.2.1
  • A ranking failure: a relevant movie placed too low in the recommendation list. section 3.2.1

Movie recommendations by preferred genre

Group users by their most-watched genre, such as action, drama, comedy, or indie, then compare error rates across groups. A much higher miss rate for indie-film users signals a subgroup quality gap hidden by aggregate performance. section 3.2.1

Figure 3.7 represents how the error rate for a specific subgroup is calculated. In this context, the numerator "number of errors for subgroups" could mean false positives, false negatives, or both, depending on what you're analyzing.

Use the calculation figure to connect subgroup error rate with the chosen numerator, such as false positives, false negatives, or both. section 3.2.1

How AI Model Evaluation covers it

How deep it goes
Defines the metric and extends it to recommendation failures, stopping short of prescribing one universal error definition.
Examples
Uses movie relevance, preferred-genre subgroups, irrelevant recommendations, missed items, and ranking failures.
What it assumes
Assumes a defined prediction task and teaches why recommendation systems must specify what counts as an error.
What you can do after
Diagnose subgroup quality gaps by defining errors and comparing their rates across recommendation slices.
How it connects
Connects error rate with precision, recall, ranking quality, false positives, false negatives, and offline evaluation.

Also explained in

Compare how 10 books explain error rate →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.