A metric hierarchy maps a broad business goal to smaller, measurable contributors. In an ML system, it helps answer a practical question: if the model improves its training loss, what evidence would show that the business goal improved too? The map provides a common basis for choosing and defending metrics during system design. section 5.2.2 section 5.0
From clicks to revenue
Suppose the company-wide goal is revenue. One product’s revenue contributes to it and depends on average purchase price, purchase frequency, and number of users. Purchase frequency is affected by conversion rate (CR), which is affected by click-through rate (CTR). This puts a fast-moving signal, CTR, on a path toward the eventual goal—but does not make a click equivalent to a purchase. section 5.2.2
Use the hierarchy when selecting metrics:
- Start with the stated end goal, then break it into stages and identify a success metric for each stage. If no hierarchy exists, draw a provisional map with product managers and data scientists or seek help from a metrics or A/B testing team. section 5.2
- Choose an offline or other proxy metric that supports useful iteration, then check its relationship to the online and business outcomes above it. CTR can be more sensitive and faster to test than CR, but its place in the hierarchy is the reason to consider it—not proof that it is sufficient. section 5.2.2 section 5.4
- Recheck that relationship as the system matures. A recommendation variant can win on engagement among internal users yet be less profitable for a wider audience; improving a lower-level metric may conflict with the goal above it. section 5.2.2
The hierarchy also helps frame choices about multi-component losses and error analysis. A spam classifier, for example, can improve its classification loss without adequately representing user happiness or harmful bursts of spam. Better error analysis alone cannot rescue a system whose metrics miss the real objective. Keep the hierarchy available as the design changes, and use production outcomes to challenge its assumed links. section 5.2.2 section 9.1.3 section 5.4
