Reweighting

As explained in Ensemble Methods for Machine Learning

Reweighting is a technique that changes the relative importance assigned to training examples during iterative learning.

Reweighting changes which training examples receive attention during iterative learning. In AdaBoost, examples classified incorrectly receive higher weights, so the next weak learner focuses on repairing earlier mistakes. section 4.2.1 section 4.6

An AdaBoost reweighting cycle proceeds as follows:

• Start with equal weights for all training examples. section 4.2.1
• Train a weak learner, such as a decision stump, using the weighted examples. section 4.6
• Increase the weights of misclassified examples before training the next learner. section 4.2.1
• Combine the diverse weak learners through a weighted additive ensemble. section 4.2.1 section 4.6

Three decision stumps

The first stump receives equally weighted examples. After its errors receive larger weights, a second stump prioritizes those cases; a third responds to the next set of mistakes. Although the individual stumps have error rates of 15%, 20%, and 25%, their combined ensemble reaches 9%. section 4.2.1

Figure 4.4 At the start of iteration 2, training examples misclassified in iteration 1 (shown with larger markers in figure 4.3, right) are assigned higher weights. This is visualized on the left, where each example’s size is proportional to its weight. Since weighted examples have higher priority, the new decision stump in the sequence (right) ensures that these are now correctly classified. Observe that the new decision stump on the right correctly classifies most of the misclassified examples (shown larger) on the left.

Use the visual progression to see how larger markers represent higher-weight examples and redirect each successive stump. section 4.2.1

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains initialization, iterative updates, ensemble effects, exponential loss, and sensitivity to outliers.

Examples

Uses three sequential decision stumps whose reweighted ensemble reduces error from individual rates to 9%.

What you can do after

Apply reweighting through AdaBoost and tune learning rate and estimator count with grid-search cross validation.

How it connects

Connects AdaBoost with weak learners, decision stumps, ensemble diversity, exponential loss, LogitBoost, and cross validation.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• AdaBoostused by · AdaBoost is an ensemble-learning algorithm that sequentially combines weak learners, adaptively reweighting training examples so that later learners emphasize those earlier learners handled poorly.
• diversityproduces · Diversity is the degree of variation among members of a set or population in relevant characteristics, perspectives, behaviors, or outcomes.
• post-processingused by · Post-processing is additional processing applied to generated or computed output after its initial production to transform, validate, filter, refine, or interpret it.
• objective functionconfigures · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
• overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.