An individual measurement is one observed metric value from one experimental observation. It is also commonly called an observation or sample. For example, one execution-cost result from a trading system, one markout-profit result from a parameter setting, or one item-view outcome can each be an individual measurement. section 2.2.1 section 4.1.2 section 8.4.2
Single values vary around an underlying expectation, so they are weak evidence for choosing between system alternatives. Replication collects many individual measurements under each condition; their mean is an aggregate measurement, which is typically closer to the expectation. Increasing the number of individual measurements narrows the aggregate's variation and reduces its standard error. section 2.2.1 section 4.1.2
Execution-cost observations
Three ASDAQ execution-cost observations are 12.276, 10.145, and 12.624 mips. Against a known simulated expectation of 12 mips, their deviations range from 0.276 to -1.855 mips. Their mean is 11.682 mips, a deviation of about -0.318 mips. The individual values expose run-to-run variation; the mean supports a more stable exchange comparison. section 2.2.1
np.random.seed(17)
measurements = np.array([trading_system("ASDAQ") for _ in range(3)])
print(measurements)
[[12.27626589 10.14537192 12.62390111]
This small simulation produces three individual execution-cost observations from repeated runs on one exchange. Re-run it to see why a single result should not decide an experimental comparison. section 2.2.1
Plan the count per condition separately from the count of conditions. In response-surface optimization, many individual profit measurements at each selected threshold make each aggregate's error bar small, while limiting the number of threshold values controls experimentation cost. Randomly assign decisions to conditions until every condition reaches its required count. section 4.1.2 section 4.2.1
Guard against misleading individual-measurement counts:
- Define the observation unit that matters for generalization. A large count of items can still represent few users or few sessions. section 8.4.2
- Randomly select measurements across relevant users, sessions, items, and other available units to mitigate narrow coverage. section 8.4.2
- Choose a minimum per-condition count to limit false positives and false negatives; small samples make aggregate errors larger and comparisons less reliable. section 8.4.2
