Expressive power
Expressive power is the capacity of a model, language, or system to represent or convey a range and complexity of functions, patterns, distinctions, or ideas.
Grokking Machine Learning
Model complexity is the degree of flexibility or parameter magnitude that a model has for representing relationships in data.
- How it is used
- Uses model complexity to balance fit and validation performance when choosing or training models.
- How deep it goes
- Measures complexity through polynomial degree and coefficient norms, then works through validation curves and regularization.
- Examples
- Uses polynomial degrees 0–10 and a roof-repair analogy to contrast models that are too simple or too complex.
- What you can do after
- Evaluate polynomial degrees with training and validation errors, then select the degree with the lowest validation error.
Read how Grokking Machine Learning explains expressive power →
Deep Learning with R, Third Edition
Representational power is the ability of a model to encode and express distinctions or patterns in data.
- How it is used
- Uses model capacity to guide architecture sizing and balance fitting against generalization.
- How deep it goes
- Explains capacity through loss curves and a model-sizing workflow, without a formula for selecting the right architecture.
- Examples
- Compares MNIST classifiers of different sizes, shuffled-label memorization, and movie-review classifiers with different capacities.
- What you can do after
- Diagnose capacity problems using training and validation behavior, then adjust model size or regularization.
Read how Deep Learning with R, Third Edition explains expressive power →
Timeless Algorithms, The Foundational Ideas
The range and complexity of relationships that a model can represent.
- How it is used
- Uses capacity to distinguish fitting training data from representing relationships that generalize.
- How deep it goes
- Defines capacity through VC dimension and shattering, then relates architectural flexibility to model behavior.
- Examples
- Illustrates capacity with linearly separated points, deep decision trees, pedestrian detection, and neural language models.
- What you can do after
- Evaluate model flexibility against validation performance, data coverage, and the risk of fitting training-specific patterns.
Read how Timeless Algorithms, The Foundational Ideas explains expressive power →
Math and Architectures of Deep Learning
The capacity of a model architecture to represent a wide range of functions or input-output relationships, including ones like XOR.
- How it is used
- Frames why neural-network architectures need nonlinearity and layered structure for complex approximation tasks.
- How deep it goes
- Defines representational capacity, explains multilayer construction and XOR, and links depth to convolutional receptive fields without deriving theorem proofs.
- Examples
- Uses dog-image variation, logical XOR truth tables, and 5×5 versus stacked 3×3 convolutions.
- What you can do after
- Evaluate/understand why a task may require a deeper or larger network and why excessive capacity can overfit.
Read how Math and Architectures of Deep Learning explains expressive power →
The Art of Code
The degree to which a programming language enables ideas to be expressed clearly and concisely without unnecessary syntactic or logical manipulation.
- How it is used
- Uses expressiveness to judge whether language features make solutions clearer, safer, and closer to the problem.
- How deep it goes
- Compares equivalent solutions using practical signals and readability tradeoffs, while treating expressiveness as difficult to measure directly.
- Examples
- Compares Java type dispatch and switch fall-through, then contrasts method references, lambdas, loops, and Kotlin scope-function chains.
- What you can do after
- Evaluate equivalent solutions for expressiveness and choose language features that improve readability rather than merely shorten code.
Deep Learning with Python, Third Edition
The capacity of a model architecture to represent more complex functions or patterns.
- How it is used
- Guides capacity increases when a model stalls without clearly overfitting, then shifts attention to generalization.
- How deep it goes
- Explains capacity changes through loss-curve comparisons and a recurrent-layer stacking procedure.
- Examples
- Compares MNIST models of increasing size and a stacked GRU for temperature forecasting.
- What you can do after
- Diagnose/debug capacity limits using training and validation behavior, and adjust layers or architecture while monitoring overfitting.
Read how Deep Learning with Python, Third Edition explains expressive power →
Sutskever's List
Model capacity is the range and complexity of functions a model can represent.
- How it is used
- Connects representational range to the practical trade-offs among depth, optimization, and generalization.
- How deep it goes
- Explains capacity through training and test errors, interpolation, and optimization trade-offs, without formal derivations.
- Examples
- Contrasts polynomial fitting with AlexNet and ResNet, and uses error curves to illustrate capacity regimes.
- What you can do after
- Evaluate how model size and training affect optimization and generalization across capacity regimes.
Build a DeepSeek Model (From Scratch)
Representational capacity is the ability of a model architecture to represent functions, patterns, or input-output relationships.
- How it is used
- Explains architectural trade-offs between specialization, memory efficiency, model size, and learnable behavior.
- How deep it goes
- Compares attention representations, parameterized teacher-student architectures, and capacity limits in temperature scaling and reinforcement learning.
- Examples
- Uses specialized attention heads, toy teacher-student networks, temperature values from 1 to 10, and model sizes from 1.5B to 671B parameters.
- What you can do after
- Evaluate how architecture and parameter count constrain specialization, probability-signal absorption, and reasoning discovery.
Read how Build a DeepSeek Model (From Scratch) explains expressive power →
Machine Learning System Design
Model capacity is the flexibility of a model class to represent functions or fit data, often related to the number of parameters.
- How it is used
- Guides model-selection and debugging decisions by relating capacity to training behavior, validation performance, and generalization.
- How deep it goes
- Explains capacity through bias, variance, loss curves, and double descent, but defers the mechanism behind double descent.
- Examples
- Uses training and validation loss curves, including rising validation loss, to illustrate overfitting and capacity control.
- What you can do after
- Diagnose/debug with it by identifying underfitting, high variance, and potential overfitting from model behavior and learning curves.
Read how Machine Learning System Design explains expressive power →
Deep Learning with PyTorch, Second Edition
A model's ability to fit a wide range of functions or memorize patterns, often increasing with parameter count and effective use of parameters.
- How it is used
- Uses expressive power to explain how neural networks represent complex relationships and why capacity affects generalization.
- How deep it goes
- Explains capacity through width, depth, parameter counts, memorization, and overfitting, but stops short of formal capacity theory.
- Examples
- Uses a face-to-age memorization failure, widening a convolutional network, parameter counting, and composed tanh units.
- What you can do after
- Evaluate how width and depth change capacity, inspect parameter counts in PyTorch, and anticipate increased overfitting risk.
Read how Deep Learning with PyTorch, Second Edition explains expressive power →
Machine Learning Algorithms in Depth
Model capacity is the range and complexity of functions that a model architecture can represent.
- How it is used
- Balances model fit and generalization by choosing capacity that avoids underfitting without overfitting.
- How deep it goes
- Explains capacity through MLP layer and unit choices, then links those choices to regularization; gives no formal function-class derivation.
- Examples
- Uses MLP layer and hidden-unit counts as capacity choices, and weight decay, early stopping, and dropout as regularizers.
- What you can do after
- Diagnose underfitting and increase MLP layers or hidden units, or address overfitting with weight decay, early stopping, or dropout.
Read how Machine Learning Algorithms in Depth explains expressive power →
Inside Deep Learning: Math, Algorithms, Models
The set of functions or behaviors that a model architecture can express.
- How it is used
- Uses capacity to reason about whether a neural network can represent a problem and how normalization relates to model design.
- How deep it goes
- Distinguishes representational capacity from what optimization learns, then connects capacity to underfitting and memorization.
- Examples
- Uses increasingly erratic functions, a catering analogy, and random-label memorization to illustrate capacity.
- What you can do after
- Evaluate whether a model can memorize a dataset by checking whether training loss falls on randomly labeled data.
Read how Inside Deep Learning: Math, Algorithms, Models explains expressive power →
Machines that Think
Model capacity is the range and complexity of patterns or functions that a computational model can represent.
- How it is used
- Frames language-model scale in terms of the variety and complexity of patterns a model can represent.
- How deep it goes
- Explains parameters, width, depth, and selective activation conceptually, while stopping short of formal measures of representational capacity.
- Examples
- Uses recurrent and LSTM models, BERT, GPT-2, and GPT-3 to illustrate changes in language-model scale.
- What you can do after
- Evaluate model scale without treating parameter count as a guarantee of capability.
Also covered in
- Rearchitecting LLMsIn early access (MEAP)
- The Complete Obsolete Guide to Generative AI
- Build AI Drug Discovery PipelinesIn early access (MEAP)
- The Quick Python Book, Fourth Edition
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- overfittingOverfitting is a modeling condition in which a model adapts too closely to particular data, including its noise or idiosyncrasies, and consequently generalizes poorly to new data.
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
Related concepts
- parameterproduced by · A parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
- cognitive complexitycontrasts with · Cognitive complexity is the mental effort required to understand code, especially as its control flow and reasoning paths become more complex.
- encoderproduced by · An encoder is a component or process that transforms input into an encoded or internal representation for subsequent processing, transmission, or decoding.
- safetyhas part · Safety is the property of avoiding specified harmful, hazardous, invalid, or otherwise forbidden outcomes under relevant conditions.
- tree depthproduced by · Tree depth is the number of hierarchical steps from a root to a node; a maximum-depth setting limits the greatest number of such steps permitted during construction or traversal.
Questions about expressive power
What is expressive power?
Expressive power is the capacity of a model, language, or system to represent or convey a range and complexity of functions, patterns, distinctions, or ideas.
Which Manning books explain expressive power?
Grokking Machine Learning, Deep Learning with R, Third Edition, Timeless Algorithms, The Foundational Ideas, Math and Architectures of Deep Learning, The Art of Code, Deep Learning with Python, Third Edition, Sutskever's List, Build a DeepSeek Model (From Scratch) and 5 more.
What should I know before learning expressive power?
Manning books that teach expressive power usually introduce language model, overfitting, objective function and parameter first.
What is the difference between expressive power and cognitive complexity?
Expressive power is the capacity of a model, language, or system to represent or convey a range and complexity of functions, patterns, distinctions, or ideas. Cognitive complexity is the mental effort required to understand code, especially as its control flow and reasoning paths become more complex.
What are the parts of expressive power?
Expressive power includes safety.
What is expressive power also called?
Expressive power is also called model capacity, expressiveness, representational capacity, expressivity or network capacity.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.












