Weak learner

As explained in Grokking Machine Learning

A weak learner is a predictive model that performs only modestly better than chance but can contribute to a strong learner when combined with other weak learners.

Suppose several friends answer the same question. Each may get some answers wrong, but a vote can give a different answer from any one friend. A weak learner plays a similar role: it is an individual predictive model trained to contribute to a combined model, called a strong learner. In these examples, weak learners are usually small decision trees. They can be used for classification or regression. section 12.1 section 12.0

How the learners are trained and combined depends on the ensemble method:

• Bagging trains learners on random samples of the data. For classification, their predictions are combined by voting; for regression, they are averaged. section 12.1
• AdaBoost trains learners in sequence. After each one, incorrectly classified points receive more attention; the learners then contribute to a score-weighted vote. section 12.3
• Gradient boosting trains later learners to predict residuals—the differences between labels and the combined prediction so far. Their predictions are added using a learning rate. section 12.4

Spam or ham?

Imagine classifying emails using counts of the words “lottery” and “sale.” One deep decision tree fits the training emails perfectly but bends its boundary to classify two isolated points. Instead, a manually built random forest trains three depth-1 trees on random subsets. Each tree supplies a simple boundary, and their votes determine the forest’s prediction. A Scikit-Learn forest with five depth-1 trees classifies the data well while treating the two troublesome points as outliers. section 12.2

Figure 12.6 The way to obtain the predictions of the random forest is by combining the predictions of the three weak learners. On the top, we can see the three boundaries of the decision trees from figure 12.5. On the bottom, we can see how the three decision trees vote to obtain the boundary of the corresponding random forest.

Compare the three shallow-tree boundaries with the boundary produced by their votes. section 12.2

Boosting gives weak learners a different job. In an AdaBoost classification example, each learner is a depth-1 tree trained on the whole dataset. Misclassified points are enlarged before the next tree is fitted. The trees receive scores based on their accuracy, so their votes need not have equal weight. A random forest, by contrast, gets its learners from random data subsets rather than repeatedly focusing on earlier mistakes. section 12.3 section 12.2

Predicting app engagement

For eight users, age is used to predict days of app use per week. A gradient-boosting model begins with a depth-0 tree that predicts the average label, 4, for everyone. It calculates each residual as the actual label minus that prediction, then fits a new tree to the residuals. With a learning rate of 0.8, an initial prediction of 4 and a second-tree prediction of 3 combine as 4 + 0.8 × 3 = 6.4. Further learners fit the remaining residuals. section 12.4

XGBoost also fits trees to residuals. Its tree-building procedure uses a similarity score to choose splits and prunes splits that fail a minimum-gain condition. Weak learners therefore need not all have the same shape or depth across methods. To see their contribution yourself, fit a shallow random forest or AdaBoost classifier to a small labeled dataset, inspect its individual trees, and compare their predictions with the ensemble’s. section 12.5 section 12.2 section 12.3

How Grokking Machine Learning covers it

How deep it goes

Works through random sampling, mistake-focused training, residual fitting, voting, and additive predictions using small decision trees.

Examples

Illustrates shallow trees on spam emails, weighted classifiers on two-class points, and residual-fitting trees on app engagement data.

What it assumes

Assumes basic familiarity with decision trees while defining learner as another name for a machine-learning model.

What you can do after

Apply it via Scikit-Learn ensemble classifiers and regressors, then inspect how multiple trees contribute to predictions.

How it connects

Connects decision trees with bagging, random forests, AdaBoost, gradient boosting, XGBoost, and strong learners.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningA weak learner is a simple predictive model that performs only slightly better than random guessing.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA predictive model that performs only slightly better than a simple baseline and is combined with other models in an ensemble.

Compare how 3 books explain weak learner →

Related concepts

• AdaBoostused by · AdaBoost is an ensemble-learning algorithm that sequentially combines weak learners, adaptively reweighting training examples so that later learners emphasize those earlier learners handled poorly.
• boostingused by · Boosting is an ensemble-learning technique that builds predictive models sequentially and combines them, with later models improving on weaknesses of earlier ones.
• decision treekind of · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
• gradient boostingused by · Gradient boosting is an ensemble-learning technique that sequentially adds predictive models, typically weak learners, to reduce a loss by fitting its gradient.
• sequential ensemble methodused by · A sequential ensemble method trains base learners in successive stages, with later learners depending on or correcting the outputs of earlier learners.
• baggingused by · Bagging, or bootstrap aggregation, is an ensemble technique that trains multiple models on bootstrap samples of data and combines their predictions.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.