Parameter count is the number of parameters available to learn statistical relationships. It represents model size and potential representational capacity, not what the model necessarily learns: that depends on the training data and compute supplied. section 13.1.2 section 13.2.1
Across the models studied, held-out loss kept declining as parameter counts rose, provided sufficient data and compute were available. This does not establish that larger models generalize reliably to every task or unfamiliar distribution. section 13.2.1 section 13.3.2
Comparing model sizes
Researchers compared more than 400 transformer language models trained for next-token prediction, measuring held-out cross-entropy as parameter count, training data, and compute varied. section 13.1.2





