liveBook · Manning

Meta-feature

As explained in Ensemble Methods for Machine Learning

A meta-feature is a feature supplied to a higher-level estimator and derived from the output of a base estimator.

A meta-feature is a new input for a higher-level estimator, produced from a base estimator’s prediction or prediction probability. It lets stacking learn how to combine diverse classifiers instead of applying a fixed weighting rule. section 3.3.1

A stacking workflow creates meta-features in stages:

  • Fit a diverse collection of level-1 base estimators on training data. section 3.3.1
  • Generate one or more meta-features per example from each estimator’s predictions or prediction probabilities. section 3.3.1
  • Train a level-2 estimator on the resulting metadata, optionally alongside the original features in blending. section 3.3.1 section 3.4.3
  • Use cross-validation to generate each training example’s meta-features from a model that did not train on that example, then retrain the base estimators on all data. section 3.3.2

Two classifiers become two learned inputs

For a two-dimensional dataset, train a 3-nearest-neighbor classifier and Gaussian Naive Bayes. For every example, record their prediction probabilities as two meta-features. A second-level classifier then learns how those confidence signals relate to the target. With six base estimators, the metadata set contains six corresponding meta-features. section 3.3.1

Blending follows a related pattern: hold out validation data, generate meta-features there, append them to the original validation features, and fit the level-2 estimator. Cross-validation is more robust to overfitting because it supplies fold-specific validation predictions rather than predictions from models trained on the same examples. section 3.3.2 section 3.4.3

How Ensemble Methods for Machine Learning covers it

How deep it goes
Explains construction, probability or prediction forms, validation procedures, and overfitting risks without developing estimator internals.
Examples
Uses 3nn and Gaussian Naive Bayes probabilities, six-estimator stacking, and validation-set augmentation for sentiment classification.
What it assumes
Assumes familiarity with classifiers and training data while establishing base estimators, validation folds, and level-2 learning through procedures.
What you can do after
Builds stacked or blended classifiers by generating prediction-based features and training a chosen level-2 estimator.
How it connects
Connects meta-features to stacking, blending, base estimators, cross-validation, prediction probabilities, and level-2 estimators.
Cover of Ensemble Methods for Machine Learning
liveBookmanning.com
Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.