How deeply does Ensemble Methods for Machine Learning cover gradient-based one-side sampling?

Explains gradient-based selection, representative sampling, and reweighting, but stops short of implementation details or formal derivation.

Gradient-based one-side sampling

As explained in Ensemble Methods for Machine Learning

Gradient-based one-sided sampling (GOSS) is a gradient-boosting sampling technique that retains examples with large gradients while subsampling and reweighting examples with small gradients.

Gradient-based one-side sampling (GOSS) speeds gradient boosting by keeping examples with large gradient magnitudes and sampling examples with small gradients. Large gradients indicate predictions that still need substantial improvement, while small-gradient examples are already better trained. Unlike random downsampling, GOSS preserves difficult examples without letting them dominate the learner. section 5.3.1

GOSS proceeds as a targeted reduction of the training set:

Compute each training example’s gradient magnitude from the current boosting iteration. section 5.3.1
Retain examples with the largest gradients because they offer the strongest immediate learning signal. section 5.3.1
Sample a fraction of small-gradient examples so correctly classified cases remain represented. section 5.3.1
Reweight the sampled small-gradient examples to compensate for their reduced representation, then continue tree learning on the resulting subset. section 5.3.1

Large-scale document retrieval

A LightGBM document-retrieval model can use GOSS when the training set is large. It reduces the number of training examples and is typically fast to train, although downsampling may reduce generalization compared with standard gradient-boosted decision trees. section 5.5.2

GOSS is one of LightGBM’s scaling mechanisms, complementing histogram-based tree learning and exclusive feature bundling: it reduces examples, while EFB reduces features. section 5.3 section 5.3.1

How Ensemble Methods for Machine Learning covers it

Cover of Ensemble Methods for Machine Learning

liveBook manning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIGradient-based one-side sampling is a sampling algorithm that retains examples with the largest gradients and randomly samples examples with smaller gradients using corrective weights.

Compare how 2 books explain gradient-based one-side sampling →

Related concepts

LightGBMused by · LightGBM is an open-source gradient-boosting framework for efficiently training tree-based machine-learning models.
gradient boostingalternative to · Gradient boosting is an ensemble-learning technique that sequentially adds predictive models, typically weak learners, to reduce a loss by fitting its gradient.
samplingkind of · Sampling is a procedure for selecting or generating observations, values, or outcomes from a larger population, signal, or probability distribution according to a specified method.
downsamplingkind of · Downsampling is the reduction of data quantity, spatial resolution, temporal resolution, or sampling rate by retaining, selecting, or aggregating fewer representative values or observations.
diversityproduces · Diversity is the degree of variation among members of a set or population in relevant characteristics, perspectives, behaviors, or outcomes.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.