Experimentation measures whether a system change improves a business metric in a live, complex environment. It supports experimental optimization: turning engineering ideas into measured decisions about code, models, trading strategies, and infrastructure settings. Offline checks can reject obviously poor changes, but production measurement decides whether a change becomes permanent. section 1.0 section 1.1.3
The central problem is counterfactual: after revenue, engagement, execution cost, or bounce rate changes, what would the metric have been without the modification? Many uncontrolled influences—from competition and user behavior to hardware, data quality, and weather—move metrics at the same time. An experiment isolates the effect of the system change while accounting for known and unknown influences. section 1.2 section 1.2.1 section 1.3
Figure 1.1 Common engineering workflow. (1) A new idea is first implemented as a code change to the system. (2) Typically, some offline evaluation is performed that rejects ideas that are expected to negatively impact business metrics. (3) The change is pushed into the production system, and business metrics are measured there, online. Accepted changes become permanent parts of the system. The whole workflow repeats, creating reliable, continuous improvement of the system.
Use the workflow as the operating loop: implement an idea, screen it offline, measure it online, retain successful changes, and repeat. section 1.0
Search-server modification
A software engineer changes the transformation portion of a search server. Offline timing on test data sets checks whether transformation is faster. The revised code then serves real users, and its bounce rate is compared with the prior version. Lower bounce rate warrants accepting the change; faster local timing alone does not establish business impact. section 1.1.3 section 1.3
A/B testing makes experimentation concrete: compare version A with version B using replicated measurements and randomization. Replication improves decision reliability, while randomization removes confounder bias. Design starts from the intended analysis, which determines the minimum measurement count needed to balance reliable decisions against time, operational effort, and exposure to a worse version. section 2.3 section 2.4
Treat experimentation as a disciplined production practice:
- Use business-metric measurements as the ultimate decision criterion; domain knowledge, model quality, and simulation can inform choices but do not replace experiments. section 1.4 section 1.3
- Account for experimentation costs: tests consume time and operational attention and can temporarily degrade performance or expose bugs. section 1.2.1 section 2.3
- Switch from A to B only when B is practically and statistically significantly better on the business metric. section 2.4
- Make design and analysis reviewable through a shared process and peer review to reduce experimenter bias. section 8.4.4
Simulation can estimate business metrics quickly and repeatedly from historical data, often with narrow error bars, but missing counterfactual details can bias it. Experiments are slower because real time cannot be rewound, yet they remain more accurate for deciding production impact. Adaptive and inference-based methods can reduce measurement cost when comparing configurations or exploring large decision spaces. section 1.3.3 section 1.2.1 section 5.0
