Squared loss measures numeric prediction error by squaring the difference between an observed value and its prediction. Squaring makes larger misses count more heavily, so minimizing the total squared error drives a regression model toward predictions close to its targets. In linear regression, this objective appears as the residual sum of squares: the model searches for feature weights whose weighted sums best approximate the target values. section 4.3.1 section 4.4
Use squared loss when fitting a numeric target with a linear regression workflow. Convert features to numeric form, including one-hot encoding for categorical values, then fit coefficients against the residual sum of squares. Missing values must be imputed before the required additions and multiplications; outliers can affect both fitting and prediction. Weakly related features can add noise, while carefully prepared features can help a linear model address nonlinear relationships. section 4.3.1 section 4.4
Squared loss also supplies a training signal beyond ordinary linear regression:
- In gradient boosting classification, a regression-style squared loss is used under the hood; its gradient is described through residuals after probabilities are transformed to logits. Each boosting iteration fits a regression model to that gradient and adjusts predictions in the steepest-descent direction. section 5.2.2
- In XGBoost regression tuning, classical squared error can be one candidate objective alongside gamma and Tweedie objectives. Hyperparameter trials can compare these choices while cross-validation evaluates MAE. section 7.2.5
Linear regression scored with RMSE
A practical regression pipeline can one-hot encode categorical columns, pass through continuous columns, fit scikit-learn’s LinearRegression, and use five-fold cross-validation. The scorer shown computes mean_squared_error with squared=False, producing RMSE rather than the squared-error value itself; this keeps the reported error on the target’s scale while the fitted linear model minimizes residual sum of squares. section 4.3.1
Listing 5.9 Testing a gradient boosting regression class
class GradientBoostingRegression(GradientBoosting):
def fit(self, X, y):
self.init = np.mean(y) #1
y_pred = self.init * np.ones((X.shape[0],))
for k in range(self.n_estimators):
gradient = self.gradient(y, y_pred)
tree = DecisionTreeRegressor(**self.params)
tree.fit(X, -gradient) #2
self.trees.append(tree)
y_pred += (
self.learning_rate * tree.predict(X)
) #3
def predict(self, X):
y_pred = self.init * np.ones((X.shape[0],))
for tree in self.trees:
y_pred += (
self.learning_rate * tree.predict(X)
) #4
return y_pred
reg = GradientBoostingRegression(n_estimators=300,
learning_rate=0.1,
max_depth=4,
min_samples_leaf=3,
random_state=0)
reg.fit(X, y)
proba = reg.predict(Xt)
plt.figure(figsize=(8, 6))
plt.hist(proba,
bins=10,
density=True,
color='blue',
alpha=0.7) #5
plt.xlabel('Predicted Probabilities')
plt.ylabel('Density')
plt.title('Histogram of Predicted Probabilities')
plt.grid(True)
plt.show()
The regression boosting implementation makes the iterative role of the loss concrete: it initializes predictions at the target mean, computes a gradient, fits each tree to the negative gradient, and adds each tree’s scaled prediction. section 5.2.2
