Randomization assigns each measurement to version A or B by chance. In an A/B test, this isolates the effect of the version change from factors such as time of day, user demographics, or data-center location, so business-metric differences support a deployment decision. section 2.0 section 2.4
The key threat is confounder bias: a factor applied differently and consistently to the versions being compared. For each trade, request, or user, choose a version independently—for example, with an even coin flip—rather than sending one version morning traffic and the other afternoon traffic. Randomization balances known and unknown time effects across versions without requiring the assignment logic to identify them first. section 2.1.2
Randomizing exchange measurements
Suppose ASDAQ and BYSE are compared on execution cost, while morning trades carry an added cost. Assigning ASDAQ to mornings and BYSE to afternoons can falsely make ASDAQ appear cheaper. Instead, loop over both periods and use np.random.randint(2) to send each trade to ASDAQ or BYSE. Both exchanges then receive a mix of morning and afternoon observations, and the measured comparison correctly finds BYSE cheaper. section 2.1.2
Listing 2.3 A randomized measurement
def randomized_measurement():
asdaq_measurement = []
byse_measurement = []
for tod in ["morning", "afternoon"]: #1
for _ in range(100): #2
if np.random.randint(2) == 0: #3
asdaq_measurement.append(trading_system_tod("ASDAQ", tod))
else:
byse_measurement.append(trading_system_tod("BYSE", tod))
return (np.array(asdaq_measurement).mean(),
np.array(byse_measurement).mean())
This implementation records each randomized trade in the measurement list for its assigned exchange, then compares the two mean costs. section 2.1.2
Use randomization as part of an experiment, not as its entire validity argument:
- Choose versions with the intended probability, commonly 0.5 for A and 0.5 for B, and log the outcome under the selected version. section 3.1.1
- Take the designed number of measurements; replication reduces variation, while randomization addresses confounder bias. section 2.0 section 2.3.4
- Keep A and B as similar as possible apart from the intended change. A refactor or alternate database bundled only with B remains a confounder despite randomized assignment. section 8.4.1
- Do not stop early or cherry-pick results, and check iid and nonstationarity assumptions; incomplete randomization can leave bias behind. section 8.7
Equal allocation can impose a production cost while an experiment runs. In an ad-serving example, randomizing equally between click models with CTRs of 0.0050 and 0.0070 yields an overall CTR near their midpoint during the test, even though model B is better. Analyze the logged per-version outcomes, then switch only when the improvement is practically and statistically significant. section 3.1.1 section 2.3.4
