Statistical error is the discrepancy between a population’s true value and an inference or estimate derived from a sample. It does not necessarily indicate a mistake: limited information makes inferential results educated guesses about unmeasured or future events. Errors cannot be eliminated entirely, but understanding their sources improves accuracy and reliability. section 5.3.1
A key form of statistical error is:
- Type 1 error, or false positive, occurs when a test rejects a true null hypothesis and reports a meaningful difference or effect that is absent or unimportant in the broader population. section 5.3.1
- False positives can produce serious consequences, including unnecessary treatment, financial stress, prosecution, imprisonment, or distorted analytics conclusions. section 5.3.1
False-positive examples
A flu test can identify infection when none exists, or a court can convict someone who did not commit a crime. Both cases reject a true negative conclusion. section 5.3.1
import numpy as np #1
from scipy import stats as st
import matplotlib.pyplot as plt
X1 = np.random.normal(loc=75.5, scale=6.2, size=500) #2
X2 = np.random.normal(loc=76.2, scale=6.5, size=500)
def t_stat(X1, X2): #3
return st.ttest_ind(X1, X2)[0]
t_values = st.bootstrap( #4
(X1, X2),
t_stat,
n_resamples=1000,
batch=50,
method="basic",
vectorized=False,
random_state=99,
)
t_crit = -st.t.ppf(q=0.95, df=49) #5
plt.hist(result.bootstrap_distribution, bins=25) #6
plt.axvline(t_crit, color="black", linestyle="dashed")
Illustrates bootstrap resampling by generating t-statistics from repeated samples and comparing them with a critical threshold. section 5.3.1
