Huber loss balances two regression priorities: it treats small residuals like squared error, but limits the influence of large residuals by switching to scaled absolute error beyond a user-selected threshold τ. It therefore offers more outlier resistance than squared error while retaining smooth behavior near zero. section 7.3.1
Use the loss by reasoning through its threshold and optimization requirements:
- Choose τ to determine where the loss changes from squared-error behavior to absolute-error behavior. section 7.3.1
- Prefer Huber loss when extreme residuals should not dominate regression training. section 7.3.1
- Use pseudo-Huber for Newton boosting, because ordinary Huber loss includes an absolute-error component and is not twice differentiable. section 7.3.1
Gradient-boosted regression
For AutoMPG regression, LightGBM can use objective='huber' with alpha controlling the switching threshold. Randomized search and five-fold cross-validation tune alpha alongside learning rate, tree depth, subsampling, and regularization; early stopping can terminate unproductive training. XGBoost instead uses objective='reg:pseudohubererror'. section 7.3.2
