Feature selection

As explained in Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

Feature sampling selects a subset of input features as candidates for modeling or a split.

Feature selection narrows a dataset to the inputs most useful for modeling. Removing irrelevant or redundant features can reduce training costs, improve interpretability, limit false signals, and sometimes improve test performance. section 2.2.3 section 6.2

A practical workflow can combine domain judgment, simple tests, and model-based checks:

• First rule out features that have no plausible relationship with the target or fail simple univariate checks such as correlation or chi-square analysis. section 2.2.3
• Then shuffle each feature and measure whether model performance changes; features whose randomization has little effect may be irrelevant or redundant. section 2.2.3
• Prioritize explicit selection for algorithms such as linear models, which use every supplied feature and can be misled by noisy inputs. section 6.2
• Treat selection as optional for tree ensembles, which can automatically ignore some irrelevant or redundant features, while still considering its benefits. section 6.2

Feature sampling inside a random forest

Random forests use a related but different form of selection: at every decision-tree split, they sample a set of features as candidate split variables. Together with bootstrapped training examples, this creates diverse trees with less-correlated predictions, which can improve the reliability and accuracy of majority-vote results. section 5.1.2

Figure 5.4 How random forests arrives at its results by combining the different data partitioning due to its decision trees thanks to majority voting

Use the random-forest diagram to compare the diverse partitions produced by sampled features with the final majority-vote prediction. section 5.1.2

How Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI covers it

How deep it goes

Explains domain filtering, univariate tests, permutation-style shuffling, and per-split sampling, but defers detailed selection tools.

Examples

Uses correlation, chi-square analysis, feature shuffling, random-forest split candidates, diverse tree partitions, and majority voting.

What it assumes

Assumes a defined target and basic model evaluation, while teaching why irrelevant and redundant inputs can harm different algorithms.

What you can do after

Apply feature selection through domain checks, univariate tests, and performance-based shuffling before training selected models.

How it connects

Connects feature selection with feature engineering, exploratory data analysis, linear models, decision trees, random forests, and deep learning.

liveBookmanning.com

Read Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning System DesignFeature selection is the process of choosing a subset of available input features for a machine learning model.
• 

  liveBookmanning.com
  Grokking Machine LearningFeature selection is a technique that chooses relevant input features and removes unnecessary or harmful ones before model training.
• 

  liveBookmanning.com
  Statistics Every Programmer NeedsFeature selection is the process of choosing a subset of input variables for use in a predictive model.
• 

  liveBookmanning.com
  Investing for ProgrammersFeature selection is the process of choosing input variables that are most relevant or useful for a predictive model.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionFeature selection is the process of choosing a subset of input features that are useful for a machine-learning task.
• 

  liveBookmanning.com
  Outlier Detection in PythonFeature selection is the process of choosing a subset of input features that are most useful for a specified analysis or prediction task.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksFeature selection is the process of identifying and retaining input variables that contribute to a model's predictions while excluding less-relevant variables.

Compare how 8 books explain feature selection →

Related concepts

• feature engineeringpart of · Feature engineering is the creation, transformation, or selection of input features from data to provide useful representations for analysis or modeling.
• inputuses · An input is a value, information, or signal supplied to a system, computation, or interface for processing or to influence its behavior.
• curse of dimensionalityaddresses · The curse of dimensionality is the set of statistical, geometric, and computational difficulties that tend to increase as the number of dimensions grows.
• L1 regularizationused by · L1 regularization is a technique that adds a penalty proportional to the sum of absolute model-parameter values, encouraging sparse parameter estimates.
• random forestused by · A random forest is an ensemble model that combines predictions from multiple decision trees built with randomized data samples, feature selections, or both.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI. The section links point to the parts of the book it draws on.