3 Heterogeneous parallel ensembles: Combining strong learners

 

This chapter covers

  • Combining base-learning models by performance-based weighting
  • Combining base-learning models with meta-learning by stacking and blending
  • Avoiding overfitting by ensembling with cross validation
  • Exploring a large-scale, real-world, text-mining case study with heterogeneous ensembles

In the previous chapter, we introduced two parallel ensemble methods: bagging and random forests. These methods (and their variants) train homogeneous ensembles, where every base estimator is trained using the same base-learning algorithm. For example, in bagging classification, all the base estimators are decision-tree classifiers.

In this chapter, we continue exploring parallel ensemble methods, but this time focusing on heterogeneous ensembles. Heterogeneous ensemble methods use different base-learning algorithms to directly ensure ensemble diversity. For example, a heterogeneous ensemble can consist of three base estimators: a decision tree, a support vector machine (SVM), and an artificial neural network (ANN). These base estimators are still trained independently of each other.

3.1 Base estimators for heterogeneous ensembles

3.1.1 Fitting base estimators

3.1.2 Individual predictions of base estimators

3.2 Combining predictions by weighting

3.2.1 Majority vote

3.2.2 Accuracy weighting

3.2.3 Entropy weighting

3.2.4 Dempster-Shafer combination

3.3 Combining predictions by meta-learning

3.3.1 Stacking

3.3.2 Stacking with cross validation

3.4 Case study: Sentiment analysis

3.4.1 Preprocessing