An ensemble model combines predictions from multiple constituent models to improve predictive accuracy, robustness, and generalization. In tree-based modeling, a random forest aggregates many decision trees rather than relying on one potentially unstable tree. section 6.0 section 6.4
A random-forest ensemble gains strength through several coordinated mechanisms:
- Trains each decision tree on a different bootstrap sample of the data through bagging. section 6.1
- Selects random feature subsets at each split, encouraging diverse and less-correlated trees. section 6.1
- Combines classification predictions by majority vote and regression predictions by averaging. section 6.0 section 6.1
- Provides feature-importance rankings based on feature selection and impurity reduction across trees. section 6.1
Loan approval by majority vote
One tree may reject an applicant when Age ≤ 30 and Income ≤ 50,000, while another may approve based on Income > 40,000 and Age > 30. A forest combines many such tree decisions and returns the class receiving the majority vote. section 6.4
>>> rf_accuracy = metrics.accuracy_score(y_test, y_pred) * 100
>>> print(f'Accuracy: {round(rf_accuracy, 0)}%')
Accuracy: 62.0%
Evaluate an ensemble by comparing test labels with predictions; the demonstrated random forest reaches 62% accuracy. section 6.4.3
