Diminishing returns
Diminishing returns is a pattern in which successive increases in an input produce progressively smaller incremental gains in an outcome.
How Large Language Models Work
Diminishing returns is the pattern in which additional investment or effort produces progressively smaller gains in an outcome.
- How it is used
- Limits claims that LLMs can improve indefinitely through repeated self-generated training.
- How deep it goes
- Explains S-curve plateaus, degraded iterative training, and hardware constraints without developing a quantitative model.
- Examples
- Uses repeated model training, transistor growth, computing performance, GPU infrastructure, and iterative technological techniques.
- What you can do after
- Evaluate claims about boundless LLM improvement by checking for new information, new techniques, and system-level constraints.
Read how How Large Language Models Work explains diminishing returns →
Grokking Statistics
Diminishing returns is a pattern in which each additional input produces a progressively smaller increase in output.
- How it is used
- Uses capability growth and data-collection costs to judge whether a system’s operating domain is too broad.
- How deep it goes
- Explains the pattern through an S-curve, inflection point, coverage targets, and rescoping.
- Examples
- Uses golf photographs, a family life story, system features, and capability coverage as examples.
- What you can do after
- Evaluate capability growth, recognize an inflection point, and rescope an overly broad operating domain.
Related concepts
- parallelismproduced by · Parallelism is the concurrent execution of multiple computational operations, often across multiple processing resources.
- scaling lawcontrasts with · A scaling law is an empirical relationship describing how a system’s properties change as its scale or resource levels vary.
- fine-tuningaddressed by · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
- accuracymeasures · Accuracy is the degree to which an output, prediction, or measurement agrees with a correct or reference outcome under a specified criterion.
- power-law relationshipproduced by · A power-law relationship is a mathematical relationship in which one quantity is proportional to another quantity raised to a fixed exponent.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.