chapter three

3 Heterogeneous Parallel Ensembles: Combining Strong Learners

 

This chapter covers

  • Combining base learning models by performance-based weighting
  • Combining base learning models with meta-learning: stacking
  • Avoiding overfitting by ensembling with cross validation
  • A large-scale, real-world text-mining case study with heterogeneous ensembles

In the previous chapter, we introduced two parallel ensemble methods: bagging and random forest. These methods (and their variants) train homogeneous ensembles, where every base estimator is trained using the same base learning algorithm. For example, in bagging classification, all the base estimators are decision tree classifiers. In this chapter, we continue exploring parallel ensemble methods, this time focusing on heterogeneous ensembles.

Heterogeneous ensemble methods use different base learning algorithms to directly ensure ensemble diversity. For example, a heterogeneous ensemble can consist of three base estimators: a decision tree, a support vector machine (SVM) and an artificial neural network. These base estimators are still trained independently of each other.

3.1          Base estimators for heterogeneous ensembles

3.1.1                       Fitting base estimators

3.1.2                       Individual predictions of base estimators

3.2          Combining predictions by weighting

3.2.1                       Majority Vote

3.2.2                       Accuracy weighting

3.2.3                       Entropy weighting

3.2.4                       Dempster-Shafer Combination

3.3          Combining predictions by meta-learning

3.3.1                       Stacking

3.3.2                       Stacking with cross validation

3.4          Case Study: Sentiment Analysis

3.4.1                       Pre-processing

3.4.2                       Dimensionality Reduction

3.4.3                       Stacking classifiers

3.5          Summary