A p-value measures how likely an observed result, or a more extreme one, would be if the specified null hypothesis were true. For correlation evaluation, it tests whether the relationship could plausibly arise by chance. section 4.1.2
A small p-value supports evidence against a no-relationship null hypothesis, but it does not measure practical importance. Large samples can make tiny correlations statistically significant, so interpret p-values with effect sizes and confidence intervals. section 4.1.5 section 4.1.2
Interpret a p-value by checking three things:
- Statistical evidence: p < 0.05 is presented as a common minimum, while p < 0.01 indicates stronger evidence. section 4.1.2
- Effect size: a correlation such as ρ = 0.15 may remain practically weak despite p < 0.001. section 4.1.5
- Uncertainty: report a confidence interval because the same point estimate can support different decisions depending on its precision. section 4.1.2
Comparing evaluation metrics
Suppose a metric agrees with human rankings at ρ = 0.15 and p < 0.001. The relationship is unlikely to be chance, but the metric barely predicts preferences. A different metric with ρ = 0.6 and p = 0.08 may deserve more data before rejection. section 4.1.5
Listing 4.1 Computing Spearman's correlation with confidence intervals
import numpy as np
from scipy.stats import spearmanr
human_scores = [4, 2, 5, 3, 1, 4, 3, 5, 2, 1]
metric_scores = [0.7, 0.3, 0.9, 0.5, 0.1, 0.8, 0.4, 0.85, 0.25, 0.15]
corr, p_value = spearmanr(human_scores, metric_scores) #A
n_bootstraps = 1000
bootstrap_corrs = []
for _ in range(n_bootstraps): #B
indices = np.random.choice(len(human_scores), len(human_scores), replace=True)
resampled_human = [human_scores[i] for i in indices]
resampled_metric = [metric_scores[i] for i in indices]
boot_corr, _ = spearmanr(resampled_human, resampled_metric)
bootstrap_corrs.append(boot_corr)
lower = np.percentile(bootstrap_corrs, 2.5) #C
upper = np.percentile(bootstrap_corrs, 97.5)
print(f"Spearman's ρ: {corr:.3f}, p-value: {p_value:.3f}")
print(f"95% CI: [{lower:.3f}, {upper:.3f}]")
Computes Spearman’s correlation and its p-value, then uses bootstrap resampling to report a 95% confidence interval. section 4.1.2








