Chinchilla-optimal models balance parameter count with training-token count according to a compute-optimal scaling relationship. The principle made data allocation a practical counterpart to choosing model size. section 6.1.7
GPT-3’s data mismatch
GPT-3 had 175 billion parameters but was trained on 300 billion tokens, or about 1.7 tokens per parameter. The Chinchilla analysis suggested either a much smaller 15-billion-parameter model or roughly 3.5 trillion training tokens. section 6.1.7
The ratio became a guide, not a settled rule. LLaMA 2 used about 30 tokens per parameter, while TinyLlama’s nearly 3,000-to-one ratio achieved only parity with similarly sized models, suggesting data cannot always make up for too few parameters. section 6.1.7
