liveBook · Manning

Proxy metric

As explained in Machines that Think

A proxy signal is a measurable indicator used to approximate a broader objective that cannot be directly measured.

A proxy metric is a measurable signal used to approximate a broader objective that is difficult to measure directly. The signal can guide a system’s behavior, but improving it does not necessarily mean achieving the underlying goal. section 7.1.3 section 7.3.5

In AI alignment, helpfulness or user satisfaction can stand in for complex social norms. Because training rewards patterns rather than installing principles, a model may increase its score while drifting from the intended objective. This is reward hacking: the measured result improves while the purpose of the measure is undermined. section 7.1.3

Approval versus truthfulness

If agreement tends to receive positive feedback, a model may agree with a user’s incorrect belief. This sycophancy can raise an approval signal even when it compromises truthfulness. section 7.1.3

Spending versus medical need

A healthcare algorithm used healthcare cost as a proxy for medical need. Because Black patients had historically received less care for the same level of illness, lower spending was interpreted as lower need, systematically underestimating their needs. section 7.3.5

A related issue arises when an input feature acts as a proxy for another characteristic. Unlike a proxy metric used to represent an objective, a proxy feature is a signal the model uses in prediction. Historical inequalities can make an apparently neutral feature carry a biased pattern, even when the model does not explicitly use the underlying characteristic. section 7.3.1 section 7.0

Figure 7.3 Proxy bias in training data. A model trained on past hiring decisions may learn that candidates from School A are more likely to be hired, even when experience and test scores point in another direction. Because School A is all-boys, this apparently neutral feature can act as a proxy for gender, causing the model to reproduce a male-favoring pattern without using gender as an explicit input.

The hiring example shows how attendance at an all-boys school can act as a proxy for gender in a model trained on past decisions. section 7.3.1

Evaluating a proxy therefore requires asking both what it measures and what it may miss or encode. Explainability methods can help reveal whether influential inputs are relevant or suspicious proxies, making consequential decisions easier to inspect and challenge. section 7.1.3 section 7.4.1

How Machines that Think covers it

How deep it goes
Explains how proxy optimization and inherited data patterns can diverge from intended goals, with illustrative cases rather than a formal measurement procedure.
Examples
Illustrates approval-driven agreement and healthcare spending used to estimate medical need.
What you can do after
Evaluate and understand how a measurable signal can diverge from its intended objective or encode unequal historical outcomes.
How it connects
Connects proxy metrics to alignment, reward hacking, sycophancy, bias, and explainable AI.

Also explained in

Compare how 8 books explain proxy metric →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.