liveBook · Manning

Generalization

As explained in Grokking Machine Learning, Second Edition

Generalization is a model's ability to make accurate predictions on data different from the examples used for training.

Generalization is a model’s ability to make accurate predictions on data it has not seen before. A rule that memorizes training examples may fit them perfectly yet fail on new cases. section 1.12 section 4.2

Studying for an exam

Imagine studying 80 of 100 questions and answering the remaining 20 without looking at the answers. The studied questions resemble training data; the unseen questions test whether the learned idea generalizes. Similarly, compare training and testing error rather than trusting training performance alone. section 4.2

Use generalization to judge whether a model has learned a useful pattern:

  • Separate a dataset into a training set for learning and a testing set for evaluation. section 4.2
  • Watch for overfitting: a highly complex model can reduce training error while increasing testing error. section 4.2
  • Prefer the model with low testing error, such as the degree-2 model among the illustrated degree-1, degree-2, and degree-10 choices. section 4.2
Figure 4.4 We can use this table to decide how complex we want our model. The columns represent the three models of degree 1, 2, and 10. The rows represent the training and the testing error. The solid circles are the training set, and the white triangles are the testing set. The errors at each point can be seen as the vertical lines from the point to the curve. The error of each model is the mean absolute error given by the average of these vertical lengths. Notice that the training error goes down as the complexity of the model increases. However, the testing error goes down and then back up as the complexity increases. From this table, we conclude that out of these three models, the best one is model 2, because it gives us a low testing error.

Compare how training and testing error change as model complexity increases. section 4.2

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Explains train-test evaluation, memorization, and complexity through polynomial examples, stopping short of implementation details.
Examples
Uses spam-classification rules, an exam analogy, and degree-1, degree-2, and degree-10 polynomial models.
What it assumes
Assumes basic familiarity with datasets and models while introducing training sets, testing sets, and unseen data.
What you can do after
Evaluate models with a testing set and select the option that generalizes with low testing error.
How it connects
Connects generalization to overfitting, training error, testing error, and model complexity.

Also explained in

Compare how 19 books explain generalization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.