A decision framework turns an experiment’s evidence into an explicit action. It states what to do when results meet, miss, or complicate predefined success criteria. section 6.3.3
Define the framework before interpreting the experiment:
- State the hypothesis: what change is expected and why. section 6.3.3
- Specify success criteria: which outcome would confirm the hypothesis. section 6.3.3
- Set response thresholds, such as rolling out a model to all users or continuing iteration before wider rollout. section 6.3.3
- When results are ambiguous, check experiment validity, exposure, primary and guardrail metrics, offline alignment, and practical effect size. section 6.6
Interpreting an unclear model test
A model experiment may improve mobile performance while doing little for desktop users, or help one market while failing in another. Treat planned segments as stronger evidence; treat post-hoc segments as follow-up hypotheses. The resulting action may be launch, iterate and retest, or stop investing in that model direction. section 6.6


