An experiment tests whether a change causes an outcome. It deliberately changes an independent variable, measures a dependent variable, and controls other conditions as far as possible. Unlike a description or correlation, its aim is to assess a cause-and-effect claim. section 3.2.3 section 3.4
An ideal controlled experiment has four features:
- Select participants from the population, potentially using random, stratified, or cluster selection. section 3.2.3
- Randomly assign selected participants to treatment and control groups. section 3.2.3
- Change the condition being tested while managing other conditions that could affect the result. section 3.2.3
- Measure the outcome in each group and compare the results. section 3.2.3 section 3.3.2
In marketing and product analytics, an A/B test applies this approach to variants of a website, application, advertisement, or workflow. Users are assigned to variants for a set duration; analysts then compare metrics such as conversions, subscriptions, visits, or time to complete a workflow. A test may also have more than two variants. section 1.1.2 section 3.3.2
Figure 3.14 Example of an A/B/C group comparison with minor changes to a website
The A/B/C website comparison illustrates how minor changes can be tested across multiple groups. section 3.3.2
Reducing abandoned shopping carts
A product team could compare three new page layouts with a control layout. Before running the test, it would use available data and stakeholder input to frame the question, specify null and alternative hypotheses, and quantify expected differences. Afterward, it could examine abandonment rates across groups and relevant cohorts, then consider a longer test if an early effect might fade. section 3.1.4 section 3.2.5
A/B testing is not the only option. Product teams may use pre/post comparisons or quasi-experimental designs when a true experiment is not possible. Study design and statistical test should fit the question and the data: for example, a between-subjects ANOVA is not reliable when its assumptions cannot be met. Keep conclusions proportionate to the control the design actually provides. section 1.1.4 section 5.1
An individual A/B test may yield a small improvement—or useful evidence that a change had no effect. Repeated tests can build knowledge about user behavior while guiding product decisions. Share the metrics, comparison, and limitations with stakeholders rather than treating a single result as certainty. section 3.3.2 section 1.1.2
