Suppose a model can use hundreds of inputs, but many repeat the same information or add little useful signal. Feature selection chooses a smaller, informative subset. It complements feature generation: create plausible inputs, test their value, then prune what does not earn its place. The goal is not simply fewer features; it is a model that meets its quality, speed, and interpretability needs. section 11.3 section 11.3.1
Why prune? Removing irrelevant or redundant inputs can reduce overfitting, training time, and serving cost. It can also make decisions easier to explain and eliminate unnecessary data sources from training and serving pipelines. But judge the result against the system’s constraints: a small accuracy gain may not justify slower predictions in a real-time application. section 11.3.2
Choose a method that fits the features and the cost of evaluation:
- Filter methods rank features without fitting the primary model. They scale well and can provide a quick first pass: retain the top-ranked inputs, or pass them to a more intensive method. Rankings can use individual properties or compare features for overlapping information. section 11.3.3
- Wrapper methods evaluate feature subsets using a model and a chosen metric. Sequential forward selection adds features from an empty set; sequential backward elimination removes them. These methods require an appropriate validation scheme nested within outer validation. section 11.3.3
- Embedded methods are a third family. Lasso regression illustrates selection through coefficients that reach zero as L1 regularization increases. section 11.3.3
Selecting photo-ranking inputs
A photo-ranking system might derive features from tags, EXIF coordinates, dates, and camera models. Trimming tags to the informative ones is one selection decision. Adding even a feature that leaves quality metrics unchanged can increase training and inference time when it lengthens a transformer-based model’s input. With a small set of these features, sequential greedy selection is suggested as a precise, though slow, option rather than applying filtering methods unchanged. section 11.5.2
Treat selection as part of learning, not as a harmless preliminary check. Repeatedly choosing features against the same validation split can bias the result; reassess validation when data sources or features change. In a training pipeline, select before training and freeze the chosen features for subsequent steps. Feature-importance analysis can then help explain which inputs contribute to predictions and where further feature work may be worthwhile. section 7.1 section 10.1.1 section 11.6
