Replication

As explained in Experimentation for Engineers

Replication is the repeated measurement of the same experimental conditions to reduce the effect of natural variation on an estimate.

Replication repeats measurements under the same experimental conditions and averages them. In an A/B test, it reduces natural measurement variation so estimates of each version’s business metric support a more reliable decision. section 2.0 section 2.2.1

A single measurement can fluctuate enough to reverse an apparent choice. Averaging multiple individual measurements produces an aggregate measurement that is typically closer to the metric’s expectation. As the number of measurements grows, aggregate estimates become less variable; standard error summarizes that remaining uncertainty. section 2.2.1

Choosing an exchange by average execution cost

Individual trades sometimes make ASDAQ appear cheaper and sometimes make BYSE appear cheaper. Averaging 100 runs instead gives costs near 12.11 versus 10.01 mips, and another run gives 11.88 versus 10.00. With 300 measurements per exchange, the estimates remain near 12 and 10, making BYSE the reliable lower-cost choice. section 2.1.2 section 2.2.1

Figure 2.3 Histogram of 1,000 aggregate measurements of ASDAQ cost overlaid on histogram of 1,000 individual measurements. Notice that the aggregate measurement histogram is narrower.

Compare the broad distribution of individual measurements with the narrower distribution of aggregate measurements to see why replication stabilizes an estimate. section 2.2.1

Use replication without confusing it with bias control:

• Choose the number of individual measurements during experiment design; the design specifies a minimum sample count. section 2.0 section 2.4
• Randomize assignment while collecting replicated measurements. Replication reduces variation, but taking conditions at different times can introduce sampling bias rather than remove it. section 2.1.2
• Decide between versions using both practical and statistical significance, not replication alone. section 2.4
• Validate findings later through independent replication, including a validating A/B test, a long reverse A/B test, or a holdout test. section 8.5

Replication also serves validation. Because experiment flaws may persist undetected, independent replication asks whether a result can be reproduced in later experiments rather than merely made more precise within one run. section 8.5 section 8.4.4

How Experimentation for Engineers covers it

How deep it goes

Works through averages, deviations, standard deviation, and standard error, then distinguishes later independent validation.

Examples

Uses simulated ASDAQ and BYSE execution costs, including 100- and 300-measurement averages and a time-of-day bias.

What it assumes

Introduces individual and aggregate measurements while assuming a business metric can be measured repeatedly.

What you can do after

Apply it to average repeated business-metric measurements, assess uncertainty, and validate results through later independent experiments.

How it connects

Connects replication to A/B testing, randomization, sampling bias, standard error, practical significance, and statistical significance.

liveBookmanning.com

Read Experimentation for Engineers on liveBook Get the book at manning.com

Related concepts

• varianceaddresses · Variance is the expected squared deviation of a random variable from its mean, measuring the spread of its distribution.
• precisionproduces · Precision is the proportion of items identified as positive, selected, or retrieved that satisfy the relevant correctness or relevance criterion.
• experimentationused by · Experimentation is the deliberate testing of an idea, intervention, or alternative by observing outcomes to gather evidence or compare effects.
• biasaddresses · Bias is a systematic deviation or tendency in data, measurements, estimates, judgments, or outputs that skews them relative to a relevant target or impartial baseline.
• prior probabilityuses · A prior probability distribution represents beliefs or assumptions about an unknown quantity before relevant observations or evidence are incorporated.
• aggregate metricproduces · An aggregate metric is a numerical summary that combines measurements collected across multiple events, requests, components, or operations.

This explainer was generated by Manning's knowledge graph from the text of Experimentation for Engineers. The section links point to the parts of the book it draws on.