When a product team asks whether changing game dynamics increases purchases, correlation from existing player data cannot answer the intervention question reliably. An A/B test assigns people to variants, applies the alternatives, and compares outcomes under controlled conditions. section 7.1.4 section 7.1.5
A practical experiment separates the intervention from the measurement:
- Select a population and randomly assign participants to two groups. section 7.1.5
- Fix the treatment condition for each group, such as low versus high side-quest engagement. section 7.1.5
- Measure the same outcome for both groups and compare their means. section 7.1.5
- Test whether the observed difference is statistically significant, while recognizing that running the experiment can impose opportunity costs. section 7.1.5
Game-dynamics experiment
Assign 1,000 players evenly to low- and high-engagement variants. The low group averages $92.99 in purchases and the high group $131.38, a difference of $38.39; the corresponding test treats the difference as significant. The result supports evaluating the policy through randomized assignment rather than observational association. section 7.1.5
Table 7.3 Example rows from the experimental data evaluating the effect of Side-Quest Engagement on In-Game Purchases
|
User ID
|
Side-Quest Engagement
|
In-Game Purchases
|
|---|---|---|
|
2828924d
|
low
|
224.39
|
|
7e7c2452
|
low
|
19.89
|
|
3ddf2915
|
low
|
221.26
|
|
10c3d883
|
high
|
93.21
|
|
c5080957
|
high
|
61.82
|
|
241c8fcf
|
high
|
188.76
|
Inspect example experimental rows showing assigned conditions alongside observed purchase outcomes. section 7.1.5
Randomized experiments are often more robust to mistaken causal assumptions than simulations, but infeasible, unethical, or impossible interventions require causal-model simulation instead. Real experiments can also create attrition, spillover, missing-data, or post-randomization-confounding problems. section 7.6.1 section 11.1
