Blending is a meta-learning ensemble technique: level-1 models generate predictions for one held-out validation set, and a level-2 model learns from those predictions. It simplifies stacking with cross-validation by using one split, reducing training cost while preserving diverse model outputs. section 3.3.2 section 3.5
Build a blended classifier procedurally:
- Split the training data into an 80% training portion and a 20% validation portion. section 3.4.3
- Train every level-1 estimator on the training portion, then predict the validation portion to create meta-features. section 3.4.3
- Append the meta-features to the validation features and train the level-2 estimator on this augmented validation set. section 3.4.3
- Use diverse base learners, such as random forests, Extra Trees, logistic regression, Bernoulli naive Bayes, and a linear SVM. section 3.4.3
Sentiment classification
A sentiment ensemble combines five heterogeneous classifiers and a neural-network meta-estimator. The validation data contains 500 original features plus five meta-features. In the reported run, training error was 7.836% and test error was 17.196%; the comparison shows that ensembling diverse classifiers improves classification performance. section 3.4.3 section 3.4.3
Listing 3.14 Blending models with a validation set
from sklearn.model_selection import train_test_split
def blend_models(level1_estimators, level2_estimator,
X, y , use_probabilities=False):
Xtrn, Xval, ytrn, yval = train_test_split(X, y,
test_size=0.2) #1
n_estimators = len(level1_estimators)
n_samples = len(yval)
Xmeta = np.zeros((n_samples, n_estimators))
for i, (model, estimator) in
enumerate(level1_estimators): #2
estimator.fit(Xtrn, ytrn)
Xmeta[:, i] = estimator.predict(Xval)
Xmeta = np.hstack([Xval, Xmeta]) #3
level2_estimator.fit(Xmeta, yval) #4
final_model = {'level-1': level1_estimators,
'level-2': level2_estimator,
'use-proba': use_probabilities}
return final_model
The implementation listing makes the split, prediction, augmentation, and meta-estimator steps directly inspectable. section 3.4.3
