Overfitting occurs when a model learns noise or accidental patterns specific to its training observations rather than relationships that persist in new data. Strong training performance can therefore conceal weak generalization. section 9.1.2 section 11.6.2
In a decision tree, adding depth can reveal useful interactions, but also creates small leaves where splits may fit chance patterns. Such trees have high variance: their structures and predictions can change substantially with the training sample. Restricting or pruning a tree can reduce this sensitivity, at the cost of limiting the relationships it captures. section 9.1.2 section 9.1.2
The same tension appears in neural networks and support vector machines. Model capacity and regularization shape how closely a model can fit training data; validation performance helps compare configurations. For neural networks, regularization can guide learning without necessarily reducing the number of parameters. section 11.6.2 section 11.6.2
To assess whether a model is fitting beyond useful patterns:
- Compare configurations using observations excluded from fitting; cross-validation can select a penalty parameter such as an SVM’s C. section 8.3.5
- Keep the test set untouched during selection, then use it to estimate how the completed model generalizes. section 8.3.5
Capacity alone does not determine overfitting. In tested language-modeling settings, larger models improved on held-out text when supported by sufficient data and compute; this does not guarantee reliability on unfamiliar tasks or distributions. section 13.3.2 section 13.3.2
































