LogitBoost is a sequential boosting method that combines weak learners while optimizing logistic loss. Unlike parallel ensembles, each learner responds to the ensemble's current errors. section 4.0 section 4.5
Its central intuition is to replace AdaBoost's exponential-loss updates with a more measured logistic-loss objective. Logistic loss penalizes incorrect predictions less harshly than exponential loss, making it less sensitive to outliers and noise. section 4.5.1
For binary classification with labels encoded as 0 and 1, each iteration proceeds as follows:
- Estimate each example's positive-class probability Pᵢ, initially using 0.5. section 4.5.3
- Compute the working response zᵢ = (yᵢ − Pᵢ) / (Pᵢ(1 − Pᵢ)), representing the probability error. section 4.5.3
- Set example weights Dᵢ = Pᵢ(1 − Pᵢ), then fit a shallow regression tree to the working responses. section 4.5.2 section 4.5.3
- Add the regressor's predictions to the ensemble margin and update the probabilities before the next iteration. section 4.5.3
This creates two important contrasts with AdaBoost: LogitBoost works with continuous prediction probabilities rather than discrete predictions, and therefore uses regression algorithms as weak learners. Its final classifier converts the additive margin into 0/1 predictions. section 4.5.2 section 4.5.3
A minimal implementation
A from-scratch implementation uses NumPy for the working response and weights, DecisionTreeRegressor(max_depth=1) for each regression stump, and an additive prediction loop. The implementation requires 0/1 training labels; prediction sums stump outputs and maps the sign to 0 or 1. section 4.5.3
Listing 4.5 LogitBoost for classification
import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import accuracy_score
from scipy.special import expit
def fit_logitboosting(X, y, n_estimators=10):
n_samples, n_features = X.shape
D = np.ones((n_samples, )) / n_samples
p = np.full((n_samples, ), 0.5) #1
estimators = []
for t in range(n_estimators):
z = (y - p) / (p * (1 - p)) #2
D = p * (1 - p) #3
h = DecisionTreeRegressor(max_depth=1) #4
h.fit(X, z, sample_weight=D)
estimators.append(h) #5
if t == 0:
margin = np.array([h.predict(X)
for h in estimators]).reshape(-1, )
else:
margin = np.sum(np.array([h.predict(X)
for h in estimators]), axis=0)
p = expit(margin) #6
return estimators
Inspect the complete Python fitting routine for working responses, probability-based weights, regression stumps, and additive updates. section 4.5.3
