liveBook · Manning

Model complexity

As explained in Timeless Algorithms, The Foundational Ideas

Model complexity is the degree of structural or parametric elaboration in a model that affects its flexibility and risk of fitting noise.

Model complexity describes how much structure or flexibility a model can use to represent data. More parameters can improve fit to observed examples, but a highly flexible model may also fit noise that will not recur. Model selection therefore weighs fit against complexity, with generalization to unseen data as a central concern. section 3.7 section 13.2.1

The trade-off appears in different forms:

  • Likelihood ratio tests, AIC, and BIC help compare plausible models by balancing fit against complexity. section 3.7
  • A soft-margin support vector machine permits some margin violations or misclassifications so unusual observations do not dictate the boundary. section 8.0
  • More neural-network parameters expand representational capacity, but capacity alone does not determine what the model learns; that also depends on training data and compute. section 13.2.1

The usual warning that greater flexibility can increase overfitting does not imply a universal complexity threshold. In the studied range, larger language models trained with sufficient data and compute continued to reduce held-out loss as parameter counts grew, following a power-law pattern. section 13.2.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains selection criteria, soft-margin behavior, and parameter capacity, while limiting scaling claims to the studied range.
Examples
Uses AIC and BIC, soft-margin support vector machines, and neural-network parameter scaling as examples.
What you can do after
Evaluate model choices by weighing fit against complexity and considering performance on unseen data.
How it connects
Connects model complexity to model selection, maximum likelihood, overfitting, support vector machines, and generalization.

Also explained in

Compare how 4 books explain model complexity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.