A language model can gain capacity by growing larger, but a useful system must also remain practical to train and run. Transformers made that growth easier: their layers have matching input and output structures, so layers can be stacked, while attention relies heavily on matrix multiplications that can run in parallel. section 9.1.1
Byte-pair encoding (BPE), attention, positional encoding, and limits on token-sequence length contribute to this scalability. Together, they helped make it feasible to train high-capacity transformers on very large text datasets. Scalability here describes a way to expand capacity and workload—not a guarantee that every added parameter improves every task. section 9.1.1 section 10.1.1
Compare size with task performance
A pandas experiment loads benchmark results, counts tasks by scaling pattern, and sorts selected tasks for inspection. Among 130 tasks classified as nonemergent, 45 had flat scaling and 27 had sublinear scaling. Flat means a larger model produced no measurable accuracy gain; sublinear means gains diminished as dataset size and parameter count increased. Test the task you care about rather than treating model size as a performance score. section 10.1.1
>>> import pandas as pd
>>> url = 'https://gitlab.com/tangibleai/nlpia2/-/raw/main/src/nlpia2'
>>> url += '/data/llm/llm-emmergence-table-other-big-bench-tasks.csv'
>>> df = pd.read_csv(url, index_col=0)
>>> df.shape #1
(211, 2)
>>> df['Emergence'].value_counts()
Emergence
linear scaling 58
flat 45 #2
PaLM 42
sublinear scaling 27
GPT-3/LaMDA 25
PaLM-62B 14
>>> scales = df['Emergence'].apply(lambda x: 'line' in x or 'flat' in x)
>>> df[scales].sort_values('Task') #3
Task Emergence
0 abstract narrative understanding linear scaling
1 abstraction and reasoning corpus flat
2 authorship verification flat
3 auto categorization linear scaling
4 bbq lite json linear scaling
.. ... ...
125 web of lies flat
126 which wiki edit flat
127 winowhy flat
128 word problems on sets and graphs flat
129 yes no black white sublinear scaling
[130 rows x 2 columns] #4
Inspect the pandas commands and category counts to see how benchmark results are grouped before drawing conclusions about scaling. section 10.1.1
A larger model can still contradict itself
In a question about the legs of two cows and two bulls, a Llama-2-7B-chat generation answers “16 legs” but then concludes “4 + 4 = 8 legs.” Its first answer is correct; its explanation is inconsistent. An earlier GPT text-generation experiment gives varied incorrect answers to the same question. These outputs are useful checks on claims that scaling alone delivers dependable reasoning. section 10.1.1 section 10.1
Scaling is also a design concern outside model training: a rule-based chatbot must be considered in terms of maintenance and scalability as conversation needs grow. For language models, the practical takeaway is similar: measure whether added capacity improves the intended work, and check the reliability of the resulting outputs. section 12.8 section 10.1.1
