liveBook · Manning

Parameter count

As explained in Timeless Algorithms, The Foundational Ideas

Model size is the scale of a model's structure or parameterization.

Parameter count is the number of parameters available to learn statistical relationships. It represents model size and potential representational capacity, not what the model necessarily learns: that depends on the training data and compute supplied. section 13.1.2 section 13.2.1

Across the models studied, held-out loss kept declining as parameter counts rose, provided sufficient data and compute were available. This does not establish that larger models generalize reliably to every task or unfamiliar distribution. section 13.2.1 section 13.3.2

Comparing model sizes

Researchers compared more than 400 transformer language models trained for next-token prediction, measuring held-out cross-entropy as parameter count, training data, and compute varied. section 13.1.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines parameter count as capacity and describes declining held-out loss, but limits the finding to tested settings and supported training conditions.
Examples
Compares more than 400 transformer language models using next-token prediction and held-out cross-entropy.
What you can do after
Evaluate how held-out loss changes with parameter count while accounting for training data and compute.
How it connects
Connects model size to training data, compute, cross-entropy loss, overfitting, and generalization.

Also explained in

Compare how 6 books explain parameter count →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.