The common workflow is an endless loop for steadily improving an engineered system: generate an idea, implement it, evaluate it offline, measure it online, and retain accepted changes. section 1.0
Run each proposed change through three stages:
- Implement the idea as a code change, using code review and unit testing as quality controls. section 1.0
- Evaluate offline with previously logged production data to estimate business metrics and reject changes expected to worsen them. section 1.0
- Measure online by pushing the change into production and observing its impact; test multiple configurations when parameters or flags require tuning. section 1.0
Ad-placement change
An engineer implements a proposed change to an advertisement system, checks it with review and tests, estimates expected clicks from logged data, and then measures production business metrics before accepting it permanently. section 1.0
Figure 1.1 Common engineering workflow. (1) A new idea is first implemented as a code change to the system. (2) Typically, some offline evaluation is performed that rejects ideas that are expected to negatively impact business metrics. (3) The change is pushed into the production system, and business metrics are measured there, online. Accepted changes become permanent parts of the system. The whole workflow repeats, creating reliable, continuous improvement of the system.
The workflow diagram makes the repeated progression from idea to implementation, offline rejection, online measurement, and accepted change explicit. section 1.0
