liveBook · Manning

Incentive mismatch

As explained in How Large Language Models Work

An incentive mismatch occurs when the objective optimized by a system differs from the behavior or outcome actually desired.

An incentive mismatch occurs when a system is trained toward an objective that differs from the behavior or outcome people actually want. Machine learning often uses a proxy problem: a more manageable target that correlates with the real goal but is easier to optimize. section 4.1.1

The mismatch appears when training rewards what is learnable rather than exactly what is desired:

  • Accuracy may be difficult to optimize directly, so training uses cross-entropy loss as a proxy. section 4.1.1
  • A model can therefore improve its training objective without improving the intended behavior, producing unexpected results or low performance. section 4.1.1
  • The consequences depend partly on how the loss function creates its incentives; gradient descent is treated as the next training component. section 4.1.1

How How Large Language Models Work covers it

How deep it goes
Defines proxy problems and cross-entropy loss, then defers the loss-function mechanics until after gradient descent.
Examples
Contrasts accuracy with cross-entropy loss as a concrete example of replacing a desired target with a learnable proxy.
What you can do after
Evaluate training objectives by asking whether the learnable proxy tracks the behavior or outcome actually desired.
How it connects
Connects proxy problems, cross-entropy loss, accuracy, loss functions, gradient descent, and machine learning.
Cover of How Large Language Models Work
liveBookmanning.com
Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

Compare how 2 books explain incentive mismatch →

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.