liveBook · Manning

Expressive power

As explained in Machines that Think

Model capacity is the range and complexity of patterns or functions that a computational model can represent.

Expressive power is the range and complexity of patterns a model can represent. In a language model, learned parameters shape this capacity by encoding constraints on how input features interact—not by literally storing sentences, rules, or facts. section 4.2.1

Width gives a model room to represent more features at each step; depth lets it combine and transform those features across layers. Together, they affect how finely the model can distinguish possible inputs. Greater capacity can preserve distinctions that a limited model would merge, raising the ceiling on patterns it can learn. section 4.2.1

Parameter count is only an approximate measure of scale, not a guarantee of capability. In large dense models, computation can become limiting before representational capacity is fully used; selective activation instead lets specialized components contribute when relevant. section 4.2.1 section 4.4

How Machines that Think covers it

How deep it goes
Explains parameters, width, depth, and selective activation conceptually, while stopping short of formal measures of representational capacity.
Examples
Uses recurrent and LSTM models, BERT, GPT-2, and GPT-3 to illustrate changes in language-model scale.
What you can do after
Evaluate model scale without treating parameter count as a guarantee of capability.
How it connects
Connects model capacity to parameters, neural network depth and width, Transformers, and selective activation.

Also explained in

Compare how 13 books explain expressive power →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.