liveBook · Manning

Model complexity

As explained in Grokking Machine Learning, Second Edition

Model complexity is a measure of how elaborate a model is, often reflected by the number and magnitudes of its coefficients.

Model complexity describes how elaborate a model is. A model tends to become more complex when it uses more features or larger coefficients. In a movie recommender, a few small, meaningful weights look simpler than many large weights that may memorize the training data. section 4.8

Two movie-recommendation models

Model 1 predicts movie 11 with 2x3 + 1.4x7 − 0.5x9 + 4. Model 2 uses all ten movie features, with coefficients ranging from −103 to 378. Model 2 appears more complex and more likely to overfit, while Model 1 suggests that movies 3, 7, and 9 are the important related features. section 4.8

Use the model equation to inspect how many features have nonzero weights and how large those weights are. section 4.8

Two common measurements are the L1 norm, the sum of absolute coefficient values, and the L2 norm, the sum of squared coefficient values. Both ignore the bias because it represents the baseline prediction rather than model complexity. For the example, Model 1 has L1 = 3.9 and L2 = 6.21, whereas Model 2 has L1 = 1,089 and L2 = 227,131. section 4.9

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Defines complexity, compares two equations, and calculates L1 and L2 norms while excluding the bias.
Examples
Uses a ten-movie recommendation dataset and two contrasting prediction equations with explicitly calculated norms.
What you can do after
Evaluate model complexity by inspecting coefficients and calculating L1 or L2 norms.
How it connects
Connects model complexity with linear regression, overfitting, coefficients, features, bias, L1 norm, and L2 norm.

Also explained in

Compare how 4 books explain model complexity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.