chapter six

6 The birth of hyperscale

 

This chapter covers

  • The scaling canon: Kaplan, Chinchilla, and post-Chinchilla scaling laws
  • Sparks of artificial general intelligence and emergent behavior
  • Smooth loss, jagged evals, and inverse scaling
  • The hidden work: Data and model parallelism
  • GPipe and pipeline parallelism
Papers
  • Scaling Laws for Neural Language Models (2020) Kaplan et al.
  • GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism (2019) Huang et al.

Modern AI has revealed that bigger models perform better. The most explicit demonstration came with the leap in performance from GPT-2 (2019) to GPT-3 (2020). GPT-2 was modest by today’s standards: it had 1.5 billion parameters and was trained on a relatively narrow dataset, Reddit. Still, it captured the public imagination with examples such as Ovid’s Unicorn (chapter 1). The piece was so convincing in tone and style that it reset expectations about machine-generated text, and almost everyone overlooked that it was cherry-picked and included oxymoronic references such as “four-horned unicorns.”

6.1 Scaling canon

6.1.1 Kaplan scaling laws

6.1.2 Does shape matter?

6.1.3 Scaling with data and model size

6.1.4 Scaling compute

6.1.5 Architecture comparisons

6.1.6 The scaling hypothesis

6.1.7 Chinchilla and post-Chinchilla

6.2 Scaling detours

6.2.1 Jaggedness

6.2.2 Contamination

6.3 Parallelism

6.3.1 GPipe

6.3.2 Results

6.3.3 Wise to pipeline?

6.4 Rounding errors