The token-to-parameter ratio counts training tokens processed per learned model parameter. Chinchilla scaling laws put the optimal ratio at roughly 20:1 for transformers of that era, challenging the preference for larger models trained on less data. section 6.1.7
GPT-3 and the data trade-off
GPT-3 had 175 billion parameters but trained on 300 billion tokens, about 1.7 tokens per parameter. The Chinchilla analysis suggested a roughly 15-billion-parameter model or about 3.5 trillion training tokens instead. LLaMA 2 later used about 30 tokens per parameter. section 6.1.7
