In 2019, GPT-2 made machine-generated prose difficult to dismiss. OpenAI’s transformer-based language model had 1.5 billion parameters and was trained on about 8 million web pages scraped from Reddit. It scaled up GPT-1 by more than tenfold in both parameters and training data, producing longer, more coherent passages than its predecessor. section 1.2
Its surprises extended beyond fluent prose. Without task-specific training, GPT-2 could generate news articles from headlines and answer reading-comprehension questions. In the much-discussed “Ovid’s Unicorn” sample, a whimsical prompt yielded several paragraphs with plausible journalistic conventions. The sample was cherry-picked, however, and included contradictions such as “four-horned unicorns.” section 1.2 section 6.0
Those capabilities also prompted a release dispute. OpenAI withheld the full-size model, citing fears of malicious use, while coverage connected convincing synthetic text to misinformation. GPT-2 became both a demonstration of progress and a test of how to communicate its risks. section 1.2
GPT-2 also marks a step toward hyperscale language models. In 2020, OpenAI followed it with GPT-3: 175 billion parameters, roughly 117 times as many, without reengineering the architecture. The comparison makes GPT-2 a useful baseline for understanding how increased model size, data, and compute changed expectations. section 6.0
