liveBook · Manning

Expressive power

Also called model capacity, expressiveness, representational capacity, expressivity, network capacity

Expressive power is the capacity of a model, language, or system to represent or convey a range and complexity of functions, patterns, distinctions, or ideas.

How 13 Manning books explain it

Grokking Machine Learning

Model complexity is the degree of flexibility or parameter magnitude that a model has for representing relationships in data.

How it is used
Uses model complexity to balance fit and validation performance when choosing or training models.
How deep it goes
Measures complexity through polynomial degree and coefficient norms, then works through validation curves and regularization.
Examples
Uses polynomial degrees 0–10 and a roof-repair analogy to contrast models that are too simple or too complex.
What you can do after
Evaluate polynomial degrees with training and validation errors, then select the degree with the lowest validation error.

Read how Grokking Machine Learning explains expressive power →

Deep Learning with R, Third Edition

Representational power is the ability of a model to encode and express distinctions or patterns in data.

How it is used
Uses model capacity to guide architecture sizing and balance fitting against generalization.
How deep it goes
Explains capacity through loss curves and a model-sizing workflow, without a formula for selecting the right architecture.
Examples
Compares MNIST classifiers of different sizes, shuffled-label memorization, and movie-review classifiers with different capacities.
What you can do after
Diagnose capacity problems using training and validation behavior, then adjust model size or regularization.

Read how Deep Learning with R, Third Edition explains expressive power →

Timeless Algorithms, The Foundational Ideas

The range and complexity of relationships that a model can represent.

How it is used
Uses capacity to distinguish fitting training data from representing relationships that generalize.
How deep it goes
Defines capacity through VC dimension and shattering, then relates architectural flexibility to model behavior.
Examples
Illustrates capacity with linearly separated points, deep decision trees, pedestrian detection, and neural language models.
What you can do after
Evaluate model flexibility against validation performance, data coverage, and the risk of fitting training-specific patterns.

Read how Timeless Algorithms, The Foundational Ideas explains expressive power →

Math and Architectures of Deep Learning

The capacity of a model architecture to represent a wide range of functions or input-output relationships, including ones like XOR.

How it is used
Frames why neural-network architectures need nonlinearity and layered structure for complex approximation tasks.
How deep it goes
Defines representational capacity, explains multilayer construction and XOR, and links depth to convolutional receptive fields without deriving theorem proofs.
Examples
Uses dog-image variation, logical XOR truth tables, and 5×5 versus stacked 3×3 convolutions.
What you can do after
Evaluate/understand why a task may require a deeper or larger network and why excessive capacity can overfit.

Read how Math and Architectures of Deep Learning explains expressive power →

The Art of Code

The degree to which a programming language enables ideas to be expressed clearly and concisely without unnecessary syntactic or logical manipulation.

How it is used
Uses expressiveness to judge whether language features make solutions clearer, safer, and closer to the problem.
How deep it goes
Compares equivalent solutions using practical signals and readability tradeoffs, while treating expressiveness as difficult to measure directly.
Examples
Compares Java type dispatch and switch fall-through, then contrasts method references, lambdas, loops, and Kotlin scope-function chains.
What you can do after
Evaluate equivalent solutions for expressiveness and choose language features that improve readability rather than merely shorten code.

Read how The Art of Code explains expressive power →

Deep Learning with Python, Third Edition

The capacity of a model architecture to represent more complex functions or patterns.

How it is used
Guides capacity increases when a model stalls without clearly overfitting, then shifts attention to generalization.
How deep it goes
Explains capacity changes through loss-curve comparisons and a recurrent-layer stacking procedure.
Examples
Compares MNIST models of increasing size and a stacked GRU for temperature forecasting.
What you can do after
Diagnose/debug capacity limits using training and validation behavior, and adjust layers or architecture while monitoring overfitting.

Read how Deep Learning with Python, Third Edition explains expressive power →

Sutskever's List

Model capacity is the range and complexity of functions a model can represent.

How it is used
Connects representational range to the practical trade-offs among depth, optimization, and generalization.
How deep it goes
Explains capacity through training and test errors, interpolation, and optimization trade-offs, without formal derivations.
Examples
Contrasts polynomial fitting with AlexNet and ResNet, and uses error curves to illustrate capacity regimes.
What you can do after
Evaluate how model size and training affect optimization and generalization across capacity regimes.

Read how Sutskever's List explains expressive power →

Build a DeepSeek Model (From Scratch)

Representational capacity is the ability of a model architecture to represent functions, patterns, or input-output relationships.

How it is used
Explains architectural trade-offs between specialization, memory efficiency, model size, and learnable behavior.
How deep it goes
Compares attention representations, parameterized teacher-student architectures, and capacity limits in temperature scaling and reinforcement learning.
Examples
Uses specialized attention heads, toy teacher-student networks, temperature values from 1 to 10, and model sizes from 1.5B to 671B parameters.
What you can do after
Evaluate how architecture and parameter count constrain specialization, probability-signal absorption, and reasoning discovery.

Read how Build a DeepSeek Model (From Scratch) explains expressive power →

Machine Learning System Design

Model capacity is the flexibility of a model class to represent functions or fit data, often related to the number of parameters.

How it is used
Guides model-selection and debugging decisions by relating capacity to training behavior, validation performance, and generalization.
How deep it goes
Explains capacity through bias, variance, loss curves, and double descent, but defers the mechanism behind double descent.
Examples
Uses training and validation loss curves, including rising validation loss, to illustrate overfitting and capacity control.
What you can do after
Diagnose/debug with it by identifying underfitting, high variance, and potential overfitting from model behavior and learning curves.

Read how Machine Learning System Design explains expressive power →

Deep Learning with PyTorch, Second Edition

A model's ability to fit a wide range of functions or memorize patterns, often increasing with parameter count and effective use of parameters.

How it is used
Uses expressive power to explain how neural networks represent complex relationships and why capacity affects generalization.
How deep it goes
Explains capacity through width, depth, parameter counts, memorization, and overfitting, but stops short of formal capacity theory.
Examples
Uses a face-to-age memorization failure, widening a convolutional network, parameter counting, and composed tanh units.
What you can do after
Evaluate how width and depth change capacity, inspect parameter counts in PyTorch, and anticipate increased overfitting risk.

Read how Deep Learning with PyTorch, Second Edition explains expressive power →

Machine Learning Algorithms in Depth

Model capacity is the range and complexity of functions that a model architecture can represent.

How it is used
Balances model fit and generalization by choosing capacity that avoids underfitting without overfitting.
How deep it goes
Explains capacity through MLP layer and unit choices, then links those choices to regularization; gives no formal function-class derivation.
Examples
Uses MLP layer and hidden-unit counts as capacity choices, and weight decay, early stopping, and dropout as regularizers.
What you can do after
Diagnose underfitting and increase MLP layers or hidden units, or address overfitting with weight decay, early stopping, or dropout.

Read how Machine Learning Algorithms in Depth explains expressive power →

Inside Deep Learning: Math, Algorithms, Models

The set of functions or behaviors that a model architecture can express.

How it is used
Uses capacity to reason about whether a neural network can represent a problem and how normalization relates to model design.
How deep it goes
Distinguishes representational capacity from what optimization learns, then connects capacity to underfitting and memorization.
Examples
Uses increasingly erratic functions, a catering analogy, and random-label memorization to illustrate capacity.
What you can do after
Evaluate whether a model can memorize a dataset by checking whether training loss falls on randomly labeled data.

Read how Inside Deep Learning: Math, Algorithms, Models explains expressive power →

Machines that Think

Model capacity is the range and complexity of patterns or functions that a computational model can represent.

How it is used
Frames language-model scale in terms of the variety and complexity of patterns a model can represent.
How deep it goes
Explains parameters, width, depth, and selective activation conceptually, while stopping short of formal measures of representational capacity.
Examples
Uses recurrent and LSTM models, BERT, GPT-2, and GPT-3 to illustrate changes in language-model scale.
What you can do after
Evaluate model scale without treating parameter count as a guarantee of capability.

Read how Machines that Think explains expressive power →

Also covered in

What to know first

Related concepts

Questions about expressive power

What is expressive power?

Expressive power is the capacity of a model, language, or system to represent or convey a range and complexity of functions, patterns, distinctions, or ideas.

Which Manning books explain expressive power?

Grokking Machine Learning, Deep Learning with R, Third Edition, Timeless Algorithms, The Foundational Ideas, Math and Architectures of Deep Learning, The Art of Code, Deep Learning with Python, Third Edition, Sutskever's List, Build a DeepSeek Model (From Scratch) and 5 more.

What should I know before learning expressive power?

Manning books that teach expressive power usually introduce language model, overfitting, objective function and parameter first.

What is the difference between expressive power and cognitive complexity?

Expressive power is the capacity of a model, language, or system to represent or convey a range and complexity of functions, patterns, distinctions, or ideas. Cognitive complexity is the mental effort required to understand code, especially as its control flow and reasoning paths become more complex.

What are the parts of expressive power?

Expressive power includes safety.

What is expressive power also called?

Expressive power is also called model capacity, expressiveness, representational capacity, expressivity or network capacity.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.