Overfitting

As explained in Math and Architectures of Deep Learning

A training failure mode in which a model fits idiosyncrasies of the training data too closely and generalizes poorly.

Overfitting occurs when a neural network uses its expressive power to memorize training instances, including anomalous details, instead of learning abstractions that generalize. It therefore achieves low loss or high accuracy on training data but performs poorly on test data or real-world inference. section 9.3 section 9.4

Figure 9.11 Overfitting: data points for a binary classifier. Points belonging to different classes are visually demarcated as squares and circles. Filled squares/circles indicate training data, and unfilled squares/circles indicate test data. There are some anomalous training data instances (filled circles in the square zone). The estimated decision boundary (solid line) has become wriggly to accommodate them, which is causing many test points (unfilled squares/circles) to be misclassified. The wiggly solid curve is an example of an overfitted decision boundary. If we had chosen a “simpler” decision boundary (dashed straight line), the two anomalous training points would have been misclassified, but the machine would have performed much better in tests.

Compare a wriggly decision boundary that accommodates anomalous training points with a simpler boundary that sacrifices a few training examples but classifies test points better. section 9.3

Recognize overfitting through these patterns:

• The model has more expressive power—such as too many perceptrons or weights—relative to the number of training instances. section 9.3
• The learned decision boundary follows every nook, bend, and kink of the training data, including noise or anomalous points. section 9.3
• Training performance is excellent while testing or real-life inference performance is poor; underfitting instead harms both training and testing performance. section 9.3 section 9.4

Regularization addresses overfitting by favoring simpler solutions. Adding a penalty such as the L2 norm of the weights discourages large parameters; dropout instead randomly turns off neurons during training, reducing reliance on a small group and mitigating co-adaptation. section 9.3.6 section 9.4

Dropout in PyTorch

A model can insert torch.nn.Dropout(p=0.2) between linear layers. During training, neurons are randomly disabled; during inference, dropout is turned off. The resulting pressure to distribute weights across neurons helps prevent memorization. section 9.3.6

How Math and Architectures of Deep Learning covers it

How deep it goes

Connects capacity, decision boundaries, symptoms, regularization, and dropout, but stops short of a full optimization procedure.

Examples

Uses anomalous binary-classification points, simpler versus wriggly boundaries, and a PyTorch dropout model.

What it assumes

Assumes neural-network training, loss, and inference while grounding the distinction between memorization and abstraction.

What you can do after

Diagnose overfitting from training-test performance and apply regularization or dropout to reduce memorization.

How it connects

Ties overfitting to neural networks, decision boundaries, underfitting, regularization, L2 regularization, dropout, and Bayesian MAP estimation.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningOverfitting is a modeling condition in which a model captures noise or idiosyncrasies in training data and generalizes poorly to new data.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionThe failure mode where performance improves on training data while degrading on unseen data because the model has adapted too specifically to the training set.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionOverfitting is a modeling condition in which performance is better on training data than on previously unseen data, including when repeated tuning adapts a model to validation data.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA failure mode where a model memorizes irrelevant details of the training data and generalizes poorly, discussed here as a risk of increased capacity.
• 

  liveBookmanning.com
  Causal Inference for Data ScienceOverfitting is the condition in which a model captures noise or idiosyncrasies in training data and consequently generalizes poorly to new data.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionOverfitting is a modeling failure in which a model relies too heavily on peculiarities of its training data and performs poorly on new data.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionOverfitting is a condition in which a model matches training data too closely and performs poorly on varied unseen data.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningOverfitting is a modeling failure in which a model fits training data too closely and does not necessarily generalize well to new data.
• 

  liveBookmanning.com
  The AI Pocket BookA failure mode in which a model becomes too specialized to its training data and generalizes poorly to new cases.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA training failure mode where the model memorizes training data too closely and generalizes poorly to new inputs.
• 

  liveBookmanning.com
  Data Without LabelsOverfitting is a modeling condition in which a supervised learning model performs well on training data but less accurately on unseen data.
• 

  liveBookmanning.com
  Investing for ProgrammersOverfitting is a modeling failure in which a model captures irrelevant variation in training features and fails to generalize reliably to new data.
• 

  liveBookmanning.com
  Generative AI in ActionOverfitting is a modeling condition in which learned patterns fit training data too closely and fail to generalize well to new examples.
• 

  liveBookmanning.com
  Machine Learning System DesignA condition in which a model fits its training data too closely and performs poorly on unseen data because it captures insignificant or non-generalizable patterns.
• 

  liveBookmanning.com
  Starting Data Analytics with Generative AI and PythonOverfitting is a modeling failure in which a model fits noise or sample-specific variation rather than generalizable underlying patterns.
• 

  liveBookmanning.com
  Demand Forecasting Best PracticesOverfitting is a modeling condition in which a model fits historical data well but fails to generalize to future or otherwise unseen data.
• 

  liveBookmanning.com
  Sutskever's ListOverfitting is a modeling failure in which a model fits training data too closely and performs poorly on unseen data.
• 

  liveBookmanning.com
  Statistics Every Programmer NeedsOverfitting is a modeling condition in which a model captures idiosyncrasies of the fitting data at the expense of performance on new data.
• 

  liveBookmanning.com
  LLMs in ProductionOverfitting is a condition in which a model becomes excessively tailored to training data and performs poorly on unseen data.
• 

  liveBookmanning.com
  Graph Neural Networks in ActionA modeling failure mode in which a model fits training data patterns too closely and generalizes poorly to validation or unseen data.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackOverfitting is a training condition in which a model fits its training data or objective too closely and generalizes poorly.
• 

  liveBookmanning.com
  Building Reliable AI SystemsA model-training outcome in which a model fits training examples too closely and generalizes poorly to new data.
• 

  liveBookmanning.com
  Financial AI in PracticeOverfitting occurs when a model captures patterns specific to its training data that do not generalize reliably to new data.

Compare how 24 books explain overfitting →

Related concepts

• regularizationaddressed by · Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
• dropoutaddressed by · Dropout is a neural-network regularization technique that randomly suppresses units, connections, or activations during training to reduce overfitting.
• early stoppingaddressed by · Early stopping is a technique that terminates an iterative process before its planned completion when a specified condition indicates that further work is unnecessary or unlikely to improve the objective.
• underfittingcontrasts with · A modeling failure in which insufficient capacity or training prevents a model from capturing relevant patterns in the data, resulting in poor performance even on training data.
• failure modekind of · A failure mode is a distinct way in which a system, component, process, or operation can fail or produce an undesirable result.
• tree pruningaddressed by · Tree pruning is the removal of decision-tree branches that fail a specified split-quality condition.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.