A scaling law describes training loss as parameters, data, and compute increase within a particular regime. When data runs out or the tokenizer, optimizer, or governing rule changes, the relationship can shift. On a log–log plot, the result is a series of straight segments joined by kinks, not one universal curve. section 6.2.1
These regime changes help explain why capabilities can seem to emerge suddenly: evaluations often use thresholds, so a small, smooth improvement in loss can turn many nearly correct answers into correct ones. Benchmarks such as MMLU, GSM8K, and BIG-bench can show steps or plateaus even when training loss changes smoothly. Curves help indicate where to spend compute; evaluations reveal which measured capabilities have changed. section 6.2.1
