Statistical significance assesses whether an observed association is unlikely under a specified null hypothesis. A low p-value supports rejecting the null hypothesis, but does not establish a strong association. section 7.2.2
Use statistical significance as one part of a results-driven analysis:
- Frame a stakeholder hypothesis, inspect the relevant questions and columns, and check missing data and outliers before testing. section 6.3
- Explore the relationship visually, then select an appropriate statistical test for the variables and hypothesis. section 7.2.1 section 7.2.2
- Report uncertainty carefully: statistical significance can support or weaken a hypothesis, while correlation size indicates association strength. section 7.2.2 section 6.3
Experience and trust in AI outputs
A heatmap suggested that more experienced developers trusted AI outputs less. Spearman’s rank correlation and Kendall’s tau found significant evidence of no link: the negative correlations were close to zero, so the association was weak. section 7.2.1 section 7.2.2
trust_exp_data = (
survey_ai_users
.dropna(subset=["YearsCodePro", "AIBen"], how="any")
.loc[survey_ai_users["AIBen"] != "No answer given", :]
)
trust_rank = (
pd.Series(
trust_exp_data["AIBen"]
.factorize(sort=True)[0]
)
.rank()
)
from scipy.stats import spearmanr, kendalltau
correlation, p_value = spearmanr(
trust_exp_data["YearsCodePro"],
trust_rank
)
print("Spearman's\n", correlation, p_value)
correlation, p_value = kendalltau(
trust_exp_data["YearsCodePro"],
trust_rank
)
print("Kendall's tau\n", correlation, p_value)
Inspect executable Python code that removes incomplete responses, ranks categorical trust answers, and calculates Spearman’s correlation and Kendall’s tau with p-values. section 7.2.2
