Diminishing returns

As explained in How Large Language Models Work

Diminishing returns is the pattern in which additional investment or effort produces progressively smaller gains in an outcome.

Diminishing returns describes a plateau: each additional investment or effort produces a smaller gain than the previous one. Technology often follows an S-curve, with rapid progress slowing as constraints accumulate. section 7.1.1 section 9.2.1

For LLM self-improvement, repeatedly generating data and retraining can reduce quality rather than improve it. Continued progress requires external information, new training data, new techniques, or more capable infrastructure. section 7.1.1 section 9.2.1

The plateau can arise from several kinds of constraint:

• More transistors do not automatically produce proportional gains in frequency, power efficiency, single-threaded performance, or total computing. section 7.1.1
• Power, data, computational capacity, GPU cost, and hosting infrastructure can limit further improvement. section 7.1.1

Moore’s law and the computing plateau

Transistors may continue doubling while overall computing performance fails to double at the same rate. The example separates growth in one measure from improvement in the complete system. section 7.1.1

Figure 9.2 The S-curve, or sigmoid, shows the classic plateau behavior: at some point, you hit diminishing returns. The counterpoint to this expressed by the iterative S-curve model is that progress continues past the plateau of diminishing returns by discovering new techniques, each represented by a new S-curve. The new techniques may start worse than the existing methods but have a higher potential to surpass them.

The S-curve figure shows how progress reaches a plateau, while an iterative S-curve model shows how new techniques can begin a fresh period of improvement. section 9.2.1

How How Large Language Models Work covers it

How deep it goes

Explains S-curve plateaus, degraded iterative training, and hardware constraints without developing a quantitative model.

Examples

Uses repeated model training, transistor growth, computing performance, GPU infrastructure, and iterative technological techniques.

What you can do after

Evaluate claims about boundless LLM improvement by checking for new information, new techniques, and system-level constraints.

How it connects

Connects S-curves, self-improvement, Moore’s law, computational limits, training data, and technological innovation.

liveBookmanning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking StatisticsDiminishing returns is a pattern in which each additional input produces a progressively smaller increase in output.

Compare how 2 books explain diminishing returns →

Related concepts

• parallelismproduced by · Parallelism is the concurrent execution of multiple computational operations, often across multiple processing resources.
• scaling lawcontrasts with · A scaling law is an empirical relationship describing how a system’s properties change as its scale or resource levels vary.
• fine-tuningaddressed by · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
• accuracymeasures · Accuracy is the degree to which an output, prediction, or measurement agrees with a correct or reference outcome under a specified criterion.
• power-law relationshipproduced by · A power-law relationship is a mathematical relationship in which one quantity is proportional to another quantity raised to a fixed exponent.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.