Weak learner

As explained in Ensemble Methods for Machine Learning

A weak learner is a simple predictive model that performs only slightly better than random guessing.

A weak learner is a simple base model whose predictions are only slightly better than random guessing. For binary classification, that means accuracy just above 50%, not a highly accurate model on its own. A shallow decision tree is a common example; a depth-1 tree is called a decision stump. Weak learners are inexpensive to train, but their value comes from being combined rather than deployed alone. section 1.5 section 4.1

Weak and strong describe a model’s complexity and standalone fit, not its permanent identity. A deep decision tree with good prediction performance is a strong learner, whereas a shallow tree with barely acceptable performance is weak. Increasing tree depth can make a classifier stronger, but excessive strength can overfit and reduce deployed prediction performance. section 1.5 section 4.1

Sequential ensembles use weak learners differently from parallel ensembles:

• Train each new base estimator after the preceding one, so it can address earlier mistakes. section 4.0 section 4.1
• Combine many individually limited models into one stronger predictor through boosting. section 4.1 section 4.2
• Contrast with bagging and random forests, which typically train stronger base estimators independently. section 4.0 section 4.2.2

AdaBoost makes weak learners useful by changing which training examples matter most. It begins with equal example weights, trains a weak learner, then raises the weights of misclassified examples and lowers those of correctly classified examples. The next learner is therefore encouraged to fix the current learner’s mistakes. AdaBoost also assigns a larger ensemble weight to a learner with lower training error, then forms predictions as a weighted additive combination of all learners. section 4.2.1 section 4.2.2

Three decision stumps, one stronger boundary

Consider three depth-1 decision trees. The first stump makes some mistakes. Those points receive higher weights before fitting the second stump, which corrects many of them but creates different errors. A third stump focuses on those errors. Combining all three stumps yields a stronger classifier than any individual stump, and the resulting decision boundary can become increasingly nonlinear as more learners are added. section 4.2.1 section 4.2.2

Figure 4.2 Decision stumps (trees of depth 1, left) are commonly used as weak learners in sequential ensemble methods such as boosting. As tree depth increases, a decision stump grows into a decision tree, becoming a stronger classifier, and its performance improves. However, it isn’t possible to arbitrarily increase the strength of classifiers as they will begin to overfit during training, which decreases their prediction performance when deployed.

Compare a depth-1 stump with deeper trees to see why low-complexity learners are suitable building blocks and why simply increasing tree depth risks overfitting. section 4.1

The same sequential role appears beyond AdaBoost. Gradient boosting trains weak regression trees on residuals—the errors between labels and current predictions—so each new tree approximates a corrective update. AdaBoost instead communicates priority through example weights. Both methods direct weak estimators toward examples that previous estimators handle poorly. section 5.2.1 section 5.6

Weak learners are not automatically safe or universally superior. AdaBoost can be sensitive to outliers because repeatedly misclassified points receive increasing attention. Its fit also depends on balancing learning rate and number of base estimators; cross-validation with grid search can select that tradeoff. Sequential methods generally favor weak, low-complexity learners, while parallel ensembles typically favor strong, independently trained learners. section 4.6 section 7.2

How Ensemble Methods for Machine Learning covers it

How deep it goes

Defines weak learners, contrasts them with strong learners, and traces their weighted training and combination in AdaBoost.

Examples

Uses depth-1 decision stumps, weighted misclassified points, a three-stump ensemble, and a make_moons classification data set.

What it assumes

Builds from binary classification, decision trees, training error, and weighted training examples while defining weak learners directly.

What you can do after

Build it from scratch as a weighted AdaBoost ensemble of decision stumps and evaluate its prediction error.

How it connects

Connects weak learners to boosting, AdaBoost, gradient boosting, decision trees, bagging, random forests, residuals, and overfitting.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningA weak learner is a predictive model that performs only modestly better than chance but can contribute to a strong learner when combined with other weak learners.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA predictive model that performs only slightly better than a simple baseline and is combined with other models in an ensemble.

Compare how 3 books explain weak learner →

Related concepts

• AdaBoostused by · AdaBoost is an ensemble-learning algorithm that sequentially combines weak learners, adaptively reweighting training examples so that later learners emphasize those earlier learners handled poorly.
• boostingused by · Boosting is an ensemble-learning technique that builds predictive models sequentially and combines them, with later models improving on weaknesses of earlier ones.
• decision treekind of · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
• gradient boostingused by · Gradient boosting is an ensemble-learning technique that sequentially adds predictive models, typically weak learners, to reduce a loss by fitting its gradient.
• sequential ensemble methodused by · A sequential ensemble method trains base learners in successive stages, with later learners depending on or correcting the outputs of earlier learners.
• baggingused by · Bagging, or bootstrap aggregation, is an ensemble technique that trains multiple models on bootstrap samples of data and combines their predictions.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.