A parameter is a learned model value—a weight or bias adjusted during training. AlexNet had 60 million parameters; parameter counts matter because they affect training difficulty, runtime cost, and overfitting risk. Neurons are computational blocks, a different measure that can be ambiguous in convolutional layers with shared parameters. section 2.5.1
Model size and training data
Chinchilla analysis found roughly 20 training tokens per parameter optimal for a transformer of that era. GPT-3 had 175 billion parameters but only 300 billion training tokens, about 1.7 per parameter; the analysis suggested a roughly 15-billion-parameter model or about 3.5 trillion tokens instead. section 6.1.7
