Model selection chooses among candidate explanations by balancing data fit against model complexity. The minimum description length (MDL) principle treats successful compression as evidence that a model has captured regularity. section 8.3
MDL adds the bits needed to describe the model and the remaining data. A highly flexible model may achieve zero training error yet lose because encoding its intricacies and noise costs more than a simpler model with residual error. section 8.3
Choosing a polynomial curve
A linear polynomial underfits curved data, while a high-degree polynomial interpolates every point but wastes bits on noise. An intermediate polynomial captures the broad trend with a shorter total description. section 8.3
Figure 8.6 A simple polynomial (left) underfits the data, a complex polynomial (middle) can achieve zero training error but wastes bits describing noise, and a tradeoff polynomial (right) identifies the broad trend without including the noise. Used with permission granted by the lead author (Peter Grünwald).
Compare underfitting, noise-fitting complexity, and the trade-off model in the polynomial example. section 8.3
