Emergent behavior

As explained in LLMs in Production

Emergent behavior is a capability or pattern that appears in a model at sufficient scale despite not being observed at smaller scales.

Emergent behavior is a capability or pattern that appears when a model becomes large enough, even though smaller models could not perform the task reliably. Large language models can suddenly solve tasks that smaller models handle no better than random, especially through zero-shot and few-shot prompting. section 2.4

In practice, scale can improve several related abilities:

• Few-shot learning: generalizing from only a few examples without task-specific fine-tuning or extensive labeled data. section 2.4
• Zero-shot performance: handling entirely novel tasks through prompting alone. section 2.4
• Language robustness: disambiguating words, interpreting varied input styles, and tolerating noisy or ungrammatical text. section 2.4

Figure 2.8 Examples of LLMs demonstrating emergent behaviors when given few-shot prompting tasks after the model scale reaches a certain size

Use the figure to compare tasks that remain difficult at smaller scales with capabilities that appear after models reach a sufficient size. section 2.4

How LLMs in Production covers it

How deep it goes

Explains the scale-dependent pattern with task comparisons and examples but does not derive its internal mechanism.

Examples

Illustrates emergent behavior with tasks that move from random-level performance to successful few-shot completion as model scale increases.

What it assumes

Assumes familiarity with language models and prompting while relating the concept to zero-shot and few-shot learning.

What you can do after

Evaluate model-scale trade-offs by recognizing when prompting capabilities may appear beyond smaller-model performance.

How it connects

Connects emergent behavior to large language models, few-shot learning, zero-shot learning, prompting, attention, and transformers.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.