Experimental optimization improves engineered systems by selecting changes and operating settings from measurements of their impact on business metrics in live production. Experiments provide the most accurate impact measurements, while domain knowledge, prediction models, and simulation supplement rather than replace them. section 1.0 section 1.4
A recurring engineering workflow turns ideas into tested system improvements:
- Implement the idea as a code change, applying code review and unit testing. section 1.0
- Evaluate its expected business impact offline using previously logged production data. section 1.0
- Push surviving changes into production and measure business metrics online. section 1.0
- Accept beneficial changes as permanent parts of the system and reject poor ideas. section 1.0
Production improvement loop
A machine-learning engineer, quantitative trader, or software engineer can repeatedly move an idea from implementation through offline screening and online business-metric measurement, creating continuous improvement in a live system. section 1.0
Figure 1.1 Common engineering workflow. (1) A new idea is first implemented as a code change to the system. (2) Typically, some offline evaluation is performed that rejects ideas that are expected to negatively impact business metrics. (3) The change is pushed into the production system, and business metrics are measured there, online. Accepted changes become permanent parts of the system. The whole workflow repeats, creating reliable, continuous improvement of the system.
The workflow figure maps the full loop from code change to offline evaluation, production measurement, acceptance, and repetition. section 1.0
