Underfitting

As explained in Grokking Machine Learning

A model whose complexity is insufficient to capture the relevant structure of its dataset.

Imagine trying to answer a difficult exam after studying only a little. You may know too little to solve new questions. Underfitting is similar: a model is too simple to learn the important structure in its dataset. section 4.0

A very simple model can perform poorly because it cannot represent the pattern in the data. Underfitting is therefore a problem of insufficient model complexity, not merely a failure to match individual observations. section 4.0 section 4.1

A line applied to curved data

Suppose a dataset resembles a downward-opening parabola. A degree-1 polynomial is a straight line, so it cannot capture the curved pattern and underfits. A degree-2 polynomial captures the pattern better, while a degree-10 polynomial may overfit by passing through noisy points. section 4.1

Diagnose underfitting by comparing errors:

• Check both training and testing or validation error. section 4.2 section 4.7
• Identify underfitting when both errors are high; the model performs poorly on data it has seen and data used for evaluation. section 4.2
• Use a model complexity graph to look for a middle ground where validation error is low rather than choosing the most complex model. section 4.4

Figure 4.5 The model complexity graph is an effective tool to help us determine the ideal complexity of a model to avoid underfitting and overfitting. In this model complexity graph, the horizontal axis represents the degree of several polynomial regression models, from 0 to 10 (i.e., the complexity of the model). The vertical axis represents the error, which in this case is given by the mean absolute error. Notice that the training error starts large and decreases as we move to the right. This is because the more complex our model is, the better it can fit the training data. The validation error, however, starts large, then decreases, and then increases again—very simple models can’t fit our data well (they underfit), whereas very complex models fit our training data but not our validation data because they overfit. A happy point in the middle is where our model neither underfits or overfits, and we can find it using the model complexity graph.

Compare how training and validation errors change as polynomial complexity increases: very simple models underfit, while an intermediate model avoids both extremes. section 4.4

Practice diagnosis

Given four models with training/testing errors of (0.1, 1.8), (0.4, 1.2), (0.6, 0.8), and (1.9, 2.3), identify the underfitting model, the overfitting model, and the best candidate. section 4.8

How Grokking Machine Learning covers it

How deep it goes

Defines insufficient complexity, demonstrates it with polynomial regression, and diagnoses it through training, testing, and validation errors.

Examples

Uses an exam analogy, polynomial models of degrees 1, 2, and 10, error comparisons, and a model-selection exercise.

What you can do after

Diagnose underfitting by comparing training with testing or validation error and select a more suitable model complexity.

How it connects

Connects underfitting to overfitting, polynomial regression, training sets, testing sets, validation sets, and model complexity graphs.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning System DesignA condition in which a model is too simple to capture important relationships between features and the target variable, resulting in poor performance on both training and new data.
• 

  liveBookmanning.com
  Generative AI in ActionUnderfitting is a modeling condition in which a model fails to learn enough structure from its training data to perform well on new examples.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionUnderfitting is a condition in which a model is too simple or insufficiently trained to perform well even on its training data.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionUnderfitting is a modeling condition in which a model is too simple or insufficiently trained to capture patterns needed for good performance.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIUnderfitting is a modeling failure in which a model is too simple to capture important patterns in the data.
• 

  liveBookmanning.com
  The AI Pocket BookA failure mode where a model is too simple to capture the training pattern and performs poorly.

Compare how 7 books explain underfitting →

Related concepts

• overfittingcontrasts with · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
• genetic algorithmaddressed by · A genetic algorithm is an evolutionary optimization method that improves a population of candidate solutions through fitness-based selection and variation, often including recombination and mutation.
• bias-variance tradeoffcontrasts with · The bias-variance tradeoff is the balance between systematic error from restrictive model assumptions and error from sensitivity to variation in sampled data.
• learning curvemeasured by · A learning curve is a representation of how performance or proficiency changes as experience, training progress, or training data increases.
• expressive poweraddresses · Expressive power is the capacity of a model, language, or system to represent or convey a range and complexity of functions, patterns, distinctions, or ideas.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.