2 A/B testing: Evaluating a modification to your system

 

This chapter covers

  • Randomizing to remove measurement bias
  • Replicating to reduce measurement variation
  • Determining how many measurements to take
  • Deciding whether to accept or reject your system change

In chapter 1, you saw that the final step in the engineer’s workflow is to measure how business metrics are impacted by a modification of your system. You do this by running an experiment on the modified production system. Experiments are the most accurate way to measure changes in business metrics.

In this chapter, you’ll learn how to run an A/B test, the simplest and most widely used type of experiment. An A/B test measures the business metric for each of version A and B. If you find that B has a better business metric, you make the modification permanent; otherwise you leave the system as is.

Figure 2.1 Three stages of an A/B test: Design, Measure, and Analyze.
02-01

An A/B test has three stages (figure 2.1):

2.1 Take an ad hoc measurement

2.1.1 Simulate the trading system

2.1.2 Compare execution costs

2.2 Take a precise measurement

2.2.1 Mitigate measurement variation with replication

2.3 Run an A/B test

2.3.1 Analyze your measurements

2.3.2 Design the A/B test

2.3.3 Measure and analyze

2.3.4 Recap of A/B test stages

Summary