Bias is systematic measurement or prediction error that consistently shifts results away from the value being estimated. Unlike random variation, which replication can reduce, bias persists when measurements are gathered under systematically different conditions. section 2.1.2 section 5.2.1
Watch for bias in both experiments and adaptive systems:
- Sampling or confounder bias arises when conditions differ between measurements or groups, such as measuring alternatives at different times. section 2.1.2 section 8.7
- Model bias arises when logged data omit counterfactual outcomes or underrepresent particular feature and action combinations. section 5.2.1
- Small-sample, optimism, and experimenter bias can distort conclusions even when the experimental design appears careful. section 8.4
- Use randomization, broad replication, peer review, holdouts, and reverse A/B tests to detect or reduce bias. section 8.7
Trading costs and time of day
Suppose BYSE is measured in the morning and ASDAQ in the afternoon. If morning trading adds 2.5 units of cost, the comparison can falsely favor ASDAQ even after averaging 100 measurements. Randomly assigning exchange measurements across morning and afternoon balances this condition and makes the comparison more reliable. section 2.1.2 section 2.1.2
In a contextual bandit, biased predictions can create a feedback loop: the policy avoids actions predicted to perform poorly, so future logs contain even less evidence about those actions. Exploration supplies missing observations and helps reduce model bias. section 5.2.1 section 5.5
Listing 2.2 A simulator that accounts for time of day
def trading_system_tod(exchange, time_of_day):
if time_of_day == "morning":
bias = 2.5 #1
elif time_of_day == "afternoon":
bias = 0 #1
return bias + trading_system(exchange) #2
Inspect the simulator that adds a morning-only cost, making the source of sampling bias explicit in executable Python. section 2.1.2
