Expressive power

As explained in Machine Learning Algorithms in Depth

Model capacity is the range and complexity of functions that a model architecture can represent.

Expressive power describes the range and complexity of functions a model architecture can represent. In practice, model capacity is adjusted to avoid underfitting while controlling overfitting: choose a model complex enough to explain the data, but prefer simpler models when they explain it well. section 7.4 section 10.1

For a multilayer perceptron, adjust capacity by changing its architecture:

• Increase the number of layers or hidden units per layer when the model underfits or has insufficient validation accuracy. section 10.1
• When overfitting, use weight decay, early stopping, or dropout to encourage simpler models or improve generalization. section 10.1

Figure 10.1 MLP architecture

Use the MLP architecture diagram to locate the layers and hidden units that can be varied to change capacity. section 10.1

How Machine Learning Algorithms in Depth covers it

How deep it goes

Explains capacity through MLP layer and unit choices, then links those choices to regularization; gives no formal function-class derivation.

Examples

Uses MLP layer and hidden-unit counts as capacity choices, and weight decay, early stopping, and dropout as regularizers.

What you can do after

Diagnose underfitting and increase MLP layers or hidden units, or address overfitting with weight decay, early stopping, or dropout.

How it connects

Connects model capacity to underfitting, overfitting, regularization, model selection, and generalization.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningModel complexity is the degree of flexibility or parameter magnitude that a model has for representing relationships in data.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionRepresentational power is the ability of a model to encode and express distinctions or patterns in data.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe capacity of a model architecture to represent a wide range of functions or input-output relationships, including ones like XOR.
• 

  liveBookmanning.com
  The Art of CodeThe degree to which a programming language enables ideas to be expressed clearly and concisely without unnecessary syntactic or logical manipulation.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionThe capacity of a model architecture to represent more complex functions or patterns.
• 

  liveBookmanning.com
  Sutskever's ListModel capacity is the range and complexity of functions a model can represent.
• 

  liveBookmanning.com
  Machine Learning System DesignModel capacity is the flexibility of a model class to represent functions or fit data, often related to the number of parameters.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA model's ability to fit a wide range of functions or memorize patterns, often increasing with parameter count and effective use of parameters.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsThe set of functions or behaviors that a model architecture can express.

Compare how 10 books explain expressive power →

Related concepts

• parameterproduced by · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
• cognitive complexitycontrasts with · Cognitive complexity is the mental effort required to understand code or a system, particularly as nesting, control-flow breaks, or investigative paths increase.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.