Bias is the systematic distance between a model’s predictions and true values. It appears when a model’s assumptions or flexibility make it miss important patterns consistently, rather than merely fluctuate across training samples. section 5.1.1 section 4.3.2
A simple linear regression can have high bias when complex real-world dynamics cannot be represented by coefficients applied to the available features. Adding transformed features, such as squared or logarithmic terms, and feature interactions can reduce that bias, but raises variance and the risk of overfitting. section 4.3.2
Bias is evaluated as part of a practical trade-off with variance:
- Subsampling for pasting can make predictions less variable, but may increase bias by leaving out important parts of the data distribution. section 5.1.1
- Averaging ensembles generally reduce variance while accepting a small increase in bias; this may not cure the already high bias of weak models. section 5.1.1
- Regularization limits linear-model complexity to improve generalization, while richer feature transformations can reduce the bias of an overly simple formulation. section 4.3.2
- Extra-trees reduce variance through more random, less-correlated trees, at the cost of higher bias because random feature choices can sacrifice predictive accuracy. section 5.1.3
Choosing between random forests and extra-trees
For a tabular classification task such as predicting whether an Airbnb listing price is above the median, extra-trees randomly choose the split feature before finding its best split point. That shortcut trains faster and lowers variance through greater tree diversity, but it can raise bias relative to a forest that evaluates candidate features for each split. section 5.1.3 section 5.6
Table 5.1 Bagging and sampling strategies (view table figure)
| Averaging strategy |
What happens to data |
Parameters for BaggingClassifier/BaggingRegressor |
|---|---|---|
| Pasting |
Training examples are sampled without replacement |
bootstrap = False max_samples < 1.0 max_features = 1.0 |
| Bagging |
Training examples are sampled with replacement (bootstrapping) |
bootstrap = True max_samples = 1.0 max_features = 1.0 |
| Random subspaces |
Features are sampled (without replacement) |
bootstrap = False max_samples = 1.0 max_features < 1.0 |
| Random patches |
Training examples and features are sampled without replacement |
bootstrap = False max_samples < 1.0 max_features < 1.0 |
Compare sampling strategies to see which ones remove training examples or features and therefore can exchange lower variance for added bias. section 5.1.1
Treat bias as a model-selection signal, not an isolated score. If predictions systematically miss relevant structure, increase useful representation or flexibility; if a variance-reduction method is considered, check whether its added bias still improves validation performance. section 5.1.1 section 4.3.2
