A t-test assesses whether the means of two groups differ statistically. Use an independent-samples t-test for separate groups, or a paired-samples t-test for values measured before and after a change within the same group. Start with the question and study design, not the test. section 4.2.1 section 2.1.3
Before treating a difference as a finding:
- Inspect the distributions, including skew and extreme outliers, and check whether the groups have comparable variances. Transformations may help approximate a normal distribution. section 4.2.1
- Plan sample size and significance threshold in advance. Power analysis estimates the sample size needed to detect a specified effect at a chosen power and alpha level. section 4.2.2
- Interpret statistical significance alongside the size and practical importance of the difference. A larger sample can make a difference statistically significant without making it more useful to a decision. section 4.2.1
Support requests before and after a search change
Compare daily Help Center visits and chat support requests during the 90 days before a search-functionality change with the 90 days after it. An independent-samples t-test is proposed to assess differences, alongside descriptive trends and an additional non-parametric test. Communicate the result in terms the stakeholders can use, rather than reporting significance alone. section 2.1.3
Practice with weather data
To compare July temperatures in Boston and New York City, visualize both distributions, check t-test assumptions, assess sample size with an a priori power analysis, and run an independent-samples t-test. Then explain to a non-specialist both what the result suggests and what it cannot establish. section 4.3.4
import numpy as np from scipy import stats as st #1 n = 0 #2 mu, sigma = 75.5, 6.2 mu2, sigma2 = 77.9, 6.5 X1 = np.random.normal(mu, sigma, n) X2 = np.random.normal(mu2, sigma2, n) #3 result = st.ttest_ind(X1, X2) print(result) #4
This Python example generates two normal samples and runs an independent-samples t-test with SciPy; the sample size is supplied after power analysis. section 4.2.2
