Model complexity

As explained in Ensemble Methods for Machine Learning

The degree of structural or functional richness a model can express, often associated with the number of adjustable or branching components.

Model complexity is a model’s capacity to fit increasingly detailed patterns. More complexity often lowers training error, but it does not guarantee better predictions on unseen data. The useful target is a balance: models that are too simple underfit, while models that are too complex can overfit. section 1.3 section 1.3.2

Complexity is expressed through model-specific controls. In a decision tree, greater maximum depth creates more partitions and increases representational power. In support vector regression, a larger C prioritizes training fit over regularization, allowing more complex behavior. These settings should be judged with both training and test scores, not training score alone. section 1.3.1 section 1.3.2

A practical way to investigate complexity is to:

• Choose a complexity control, such as max_depth for a decision tree or C for an SVM. section 1.3.1 section 1.3.2
• Train models across a range of settings using repeated train/test partitions. section 1.3.1
• Compare mean training and test R² scores; rising training performance without test improvement signals limited value from extra complexity. section 1.3.1 section 1.3.2
• Select complexity for generalization rather than nearly perfect training fit. section 1.3.2

Deeper trees can stop helping

On Friedman-1 regression data, increasing decision-tree depth drives training R² toward 1. Yet test scores remain roughly consistent beyond max_depth=4, so a depth-8 tree fits the training set better without meaningfully improving predictions on new data. section 1.3.1

Figure 1.4 Comparing decision trees of different depths on the Friedman-1 regression data set using R² as the evaluation metric. Higher R² scores mean that the model achieves lower error and fits the data better. An R² score close to 1 means that the model achieves nearly zero error. It’s possible to fit the training data nearly perfectly with very deep decision trees, but such overly complex models actually overfit the training data and don’t generalize well to future data, as evidenced by the test scores.

Compare the diverging training and test curves as tree depth increases: near-perfect training fit can coincide with stagnant test performance. section 1.3.1

Complexity also matters inside ensembles. Base trees can be made more complex through maximum depth, leaf-node limits, or impurity criteria. Highly complex trees are unstable, so an ensemble may need more estimators to reduce their variability; deeper trees also cost more to train and may produce little predictive gain. section 2.5.2

How Ensemble Methods for Machine Learning covers it

How deep it goes

Works through validation comparisons for tree depth and SVM C, then extends complexity tradeoffs to ensemble base learners.

Examples

Compares tree depths and SVM C values on Friedman-1 regression data using repeated splits and R² scores.

What it assumes

Reinforces labeled supervised learning, regression, training/test partitions, and R² while contrasting them with unsupervised tasks.

What you can do after

Apply it via scikit-learn validation curves to select complexity using training and test performance.

How it connects

Connects model complexity to overfitting, underfitting, regularization, validation curves, decision trees, support vector regression, and ensemble methods.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• biascontrasts with · Bias is a systematic deviation or tendency in data, measurements, estimates, judgments, or outputs that skews them relative to a relevant target or impartial baseline.
• variancecontrasts with · Variance is the expected squared deviation of a random variable from its mean, measuring the spread of its distribution.
• l2 normmeasured by · The L2 norm is the nonnegative Euclidean magnitude of a vector, equal to the square root of the sum of its components’ squares, and to absolute value for a scalar.
• overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.