A p-value measures how likely it is to observe an association at least as extreme as the one measured, assuming the null hypothesis is true. For a conditional-independence test, it asks how probable that association would be if X and Y were independent given Z. It measures statistical evidence, not the probability that the null hypothesis itself is true. section 2.3.3
Use a p-value as one step in testing a relationship:
- Specify the null hypothesis, such as conditional independence between X and Y given Z. section 2.3.3
- Compare the p-value with a chosen significance threshold, commonly .05, .01, or .1. section 4.4.3
- Treat a value below the threshold as evidence favoring dependence and a value above it as evidence favoring conditional independence; neither result proves the underlying fact. section 2.3.3 section 4.4.3
Sample size can reverse a conclusion
For the conditional independence relation E ⊥ T | O, R, a dataset of 30 observations produced a p-value of 0.56 even though the relation was true in simulation. Bootstrap analysis repeatedly resamples different data sizes, estimates mean p-values and 90% intervals, and tracks how often the test favors independence at α = .05. As the dataset grows, expected p-values decrease, potentially crossing the threshold and causing rejection of a true independence relation. section 4.4.3
Listing 4.8 Bootstrap analysis of sensitivity of test of E ⊥ T | O, R to sample size
from numpy import mean, quantile
def sample_p_val(data_size, data, alpha): #1
bootstrap_data = data.sample(n=data_size, replace=True) #1
result = chi_square( #1
X="E", Y="T", Z=["O", "R"], #1
data=bootstrap_data, #1
boolean=False, #1
significance_level = alpha #1
) #1
p_val = result[1] #1
return p_val #1
def estimate_p_val(data_size, data=fulldata, boot_size=1000, α=.05): #2
samples = [ #2
sample_p_val(data_size, data=fulldata, alpha=α) #2
for _ in range(boot_size) #2
] #2
positive_tests = [p_val > significance for p_val in samples] #3
prob_conclude = mean(positive_tests) #4
p_estimate = mean(samples) #4
quantile_05, quantile_95 = quantile(samples, [.05, .95]) #5
lower_error = p_estimate - quantile_05 #5
higher_error = quantile_95 - p_estimate #5
return p_estimate, lower_error, higher_error, prob_conclude
data_size = range(30, 1000, 20) #6
result = list(zip(*[estimate_p_val(size) for size in data_size])) #6
Shows Python-oriented bootstrap code for sampling p-values, estimating their mean and confidence interval, and measuring the probability of favoring independence across data sizes. section 4.4.3
Causal-model validation requires extra caution. Traditional tests normally use independence as the null hypothesis, whereas a causal DAG often predicts independence and treats observed dependence as possible sampling noise. Using the traditional test in that setting is therefore a heuristic rather than a fully rigorous refutation procedure, especially because p-values change with sample size. section 4.4.2 section 4.4.3
