Statistical significance means an observed result would be unlikely by chance if the null hypothesis were true. In evaluation, a p-value below 0.05 is commonly treated as significant, while 0.01 indicates stronger evidence. section 4.1.2
Interpret significance alongside uncertainty and practical magnitude:
- Report confidence intervals with correlations; a point estimate without uncertainty can mislead. section 4.1.5
- Treat p-values as evidence against chance, not as measures of practical importance. section 4.1.5
- Check the magnitude of ρ, τ, or κ: large samples can make trivial correlations significant, while small samples can conceal meaningful agreement. section 4.1.5
A significant result is not necessarily useful
A metric with Spearman’s ρ = 0.15 and p < 0.001 is statistically significant, yet barely predicts human preferences. A metric with ρ = 0.6 and p = 0.08 may warrant more data rather than immediate rejection. section 4.1.5
Listing 4.1 Computing Spearman's correlation with confidence intervals
import numpy as np
from scipy.stats import spearmanr
human_scores = [4, 2, 5, 3, 1, 4, 3, 5, 2, 1]
metric_scores = [0.7, 0.3, 0.9, 0.5, 0.1, 0.8, 0.4, 0.85, 0.25, 0.15]
corr, p_value = spearmanr(human_scores, metric_scores) #A
n_bootstraps = 1000
bootstrap_corrs = []
for _ in range(n_bootstraps): #B
indices = np.random.choice(len(human_scores), len(human_scores), replace=True)
resampled_human = [human_scores[i] for i in indices]
resampled_metric = [metric_scores[i] for i in indices]
boot_corr, _ = spearmanr(resampled_human, resampled_metric)
bootstrap_corrs.append(boot_corr)
lower = np.percentile(bootstrap_corrs, 2.5) #C
upper = np.percentile(bootstrap_corrs, 97.5)
print(f"Spearman's ρ: {corr:.3f}, p-value: {p_value:.3f}")
print(f"95% CI: [{lower:.3f}, {upper:.3f}]")
Compute Spearman’s correlation, its p-value, and a bootstrap 95% confidence interval with Python and SciPy. section 4.1.2








