Weak learner

As explained in Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

A predictive model that performs only slightly better than a simple baseline and is combined with other models in an ensemble.

A weak learner is a simple, fast-to-train model that performs better than random guessing or only slightly better than a basic baseline. Its main value appears when many such models are combined into an ensemble. section 5.1.1 section 5.2

Weak learners take different roles in ensemble methods:

• Averaging combines predictions from multiple models, often trading a small increase in bias for lower variance. section 5.1.1
• Bagging can combine decision trees, while earlier averaging approaches commonly used simple models such as linear regression or k-nearest neighbors. section 5.1.1
• AdaBoost fits weak learners sequentially, giving more weight to incorrectly predicted observations before combining results by voting or averaging. section 5.2
• Gradient boosting adds tree models one at a time, with each tree correcting errors left by the previous ensemble. section 5.2

Choosing more weak learners in XGBoost

Increasing XGBoost’s n_estimators adds more decision trees and can improve predictions, but it also increases training time. The useful number of trees depends on the learning rate and the problem’s performance on unseen data. section 5.4.1

Figure 5.5 How gradient descent works with boosted trees

Use the figure to see how successive boosted trees reduce residual error through gradient-descent-style corrections. section 5.2

How Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI covers it

How deep it goes

Explains weak learners through bias, variance, sequential correction, and weighting, but does not derive their mathematics.

Examples

Uses linear regression, k-nearest neighbors, decision trees, AdaBoost, and XGBoost’s n_estimators parameter as concrete examples.

What you can do after

Apply via XGBoost by balancing the number of weak learners, learning rate, predictive performance, and training time.

How it connects

Connects weak learners to ensemble learning, bagging, AdaBoost, gradient boosting, decision trees, linear regression, and k-nearest neighbors.

liveBookmanning.com

Read Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningA weak learner is a predictive model that performs only modestly better than chance but can contribute to a strong learner when combined with other weak learners.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningA weak learner is a simple predictive model that performs only slightly better than random guessing.

Compare how 3 books explain weak learner →

Related concepts

• AdaBoostused by · AdaBoost is an ensemble-learning algorithm that sequentially combines weak learners, adaptively reweighting training examples so that later learners emphasize those earlier learners handled poorly.
• boostingused by · Boosting is an ensemble-learning technique that builds predictive models sequentially and combines them, with later models improving on weaknesses of earlier ones.
• decision treekind of · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
• gradient boostingused by · Gradient boosting is an ensemble-learning technique that sequentially adds predictive models, typically weak learners, to reduce a loss by fitting its gradient.
• sequential ensemble methodused by · A sequential ensemble method trains base learners in successive stages, with later learners depending on or correcting the outputs of earlier learners.
• baggingused by · Bagging, or bootstrap aggregation, is an ensemble technique that trains multiple models on bootstrap samples of data and combines their predictions.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI. The section links point to the parts of the book it draws on.