liveBook · Manning

Overfitting

As explained in Timeless Algorithms, The Foundational Ideas

Overfitting occurs when a model fits its training observations too closely without capturing relationships that remain useful beyond them.

Overfitting occurs when a model learns noise or accidental patterns specific to its training observations rather than relationships that persist in new data. Strong training performance can therefore conceal weak generalization. section 9.1.2 section 11.6.2

In a decision tree, adding depth can reveal useful interactions, but also creates small leaves where splits may fit chance patterns. Such trees have high variance: their structures and predictions can change substantially with the training sample. Restricting or pruning a tree can reduce this sensitivity, at the cost of limiting the relationships it captures. section 9.1.2 section 9.1.2

The same tension appears in neural networks and support vector machines. Model capacity and regularization shape how closely a model can fit training data; validation performance helps compare configurations. For neural networks, regularization can guide learning without necessarily reducing the number of parameters. section 11.6.2 section 11.6.2

To assess whether a model is fitting beyond useful patterns:

  • Compare configurations using observations excluded from fitting; cross-validation can select a penalty parameter such as an SVM’s C. section 8.3.5
  • Keep the test set untouched during selection, then use it to estimate how the completed model generalizes. section 8.3.5

Capacity alone does not determine overfitting. In tested language-modeling settings, larger models improved on held-out text when supported by sufficient data and compute; this does not guarantee reliability on unfamiliar tasks or distributions. section 13.3.2 section 13.3.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines overfitting and explains its causes through tree depth, sample variance, model capacity, and regularization.
Examples
Uses deep tree leaves, neural-network capacity, SVM penalty selection, and language-model performance on held-out text.
What you can do after
Diagnose overfitting by comparing validation performance and preserve an untouched test set for final evaluation.
How it connects
Connects overfitting to decision trees, variance, regularization, cross-validation, and generalization.

Also explained in

Compare how 33 books explain overfitting →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.