A business metric is a quantitative measure used to decide whether a change improves an engineered product or system. Revenue, daily active users (DAU), time spent, clicks, comments, profit, and risk can all serve as metrics when they represent the actual objective. A metric is not merely a convenient number: it is the acceptance criterion for making a production change permanent. section 1.0 section 7.0
Measure metrics online with experiments because live systems have too many interacting influences to infer a change’s impact reliably from intuition alone. Software, data and model quality, users, competitors, weather, timing, and other conditions can affect an outcome. Offline models, domain knowledge, and simulations can cheaply screen ideas, but they cannot supply the counterfactual behavior produced after deployment. section 1.3 section 1.3.2 section 1.3.3
Model quality is not product value
A short-video product improves a model’s out-of-sample cross-entropy for predicting swipes. DAU initially rises, then rises more slowly, and eventually does not improve. Cross-entropy measures model fit; DAU measures a product outcome. Improving the former does not guarantee improving the latter, so evaluate the deployed product against the business metric rather than treating a technical proxy as the goal. section 7.1.1 section 7.1
Choose metrics that are specific to the business rather than generic to an industry or model. A video service might optimize positive comments per video, time spent per day, or advertising revenue depending on its goals. Define them with stakeholders whose interests are affected, including users, investors, team members, regulators, advertisers, and dependent teams. section 7.2.1
Manage metrics as operational decision rules:
- Return to the business metric after each experiment, then reconsider the full range of product changes rather than only refining the last model or feature. section 7.1.2
- Reevaluate metrics as stakeholder interests, product maturity, user needs, and external conditions change; quarterly review is a common cadence. section 7.2.2
- Use multiple metrics when objectives compete, and specify thresholds for each metric before accepting a new system version. section 7.3 section 7.4
- Treat measurement cost and metric timescale as design constraints when selecting what to optimize and how quickly it can be evaluated. section 7.2.2
Figure 1.1 Common engineering workflow. (1) A new idea is first implemented as a code change to the system. (2) Typically, some offline evaluation is performed that rejects ideas that are expected to negatively impact business metrics. (3) The change is pushed into the production system, and business metrics are measured there, online. Accepted changes become permanent parts of the system. The whole workflow repeats, creating reliable, continuous improvement of the system.
Follow the engineering loop from implementation and offline screening to online metric measurement, acceptance, and repeated improvement. section 1.0
