Suppose players who take more side quests also spend more on in-game purchases. Is the observed difference large enough to distinguish from sampling variation? A hypothesis test compares a statistic calculated from data with a null hypothesis. For the purchase comparison, the null says the population difference in mean purchases is zero; the alternative says it is not zero. A p-value measures how surprising a result at least this strong would be under the null, not the probability that the null is true. section 7.1.3 section 7.4.5
The question and the data determine what is being tested:
- For a group comparison, estimate an effect size from sample averages and test whether it differs significantly from zero. In the game data, mean purchases are about $112 for highly engaged players and $73 for less engaged players. section 7.1.3
- For a causal DAG, test a conditional independence it implies. If the graph says education and transportation are independent given occupation and residence, the conventional test sets that independence as the null. A p-value below a chosen threshold such as .05 favors dependence, contradicting that implication. section 4.3.2
- For an estimated causal effect, test an interventional effect size rather than an observational association. Treatment–control experiments test an estimate of the average treatment effect; significance alone does not turn an observational difference into a causal effect. section 7.4.5
Testing a DAG implication in Python
With columns E (education), T (transportation), O (occupation), and R (residence), run `chi_square(X="E", Y="T", Z=["O", "R"], data=data, boolean=False, significance_level=.05)` from `pgmpy.estimators.CITests`. The inputs identify the two variables and the variables held conditional; the test evaluates the proposed independence. Running it on both a 30-row subset and the full dataset helps reveal how sample size affects the result. section 4.3.2
Interpret a test in light of its purpose. A conventional conditional-independence test places independence in the null, even when the goal is to support a DAG that predicts independence. A large p-value therefore does not establish that the DAG is right; using such tests for DAG refutation is a heuristic, not a rigorous confirmation procedure. Tests can also probe an effect estimate’s assumptions: DoWhy’s simulation-based refuters use a null that the assumptions are not refuted, and a p-value below .05 indicates refutation. section 4.4.2 section 11.5
