Multicollinearity occurs when predictor features carry strongly overlapping information. In tabular data, it often signals duplicated columns, a direct causal relationship, or multiple measurements shaped by an unrecorded latent factor. It adds computation and can impair learning-algorithm convergence, model quality, and interpretability. section 2.2.2 section 2.4.3
For numeric features, inspect pairwise correlations in a correlation matrix or list only pairs above a threshold when there are many columns. Correlations with absolute values around 0.98 to 0.99 warrant investigation; a value near minus one is also collinearity. For categorical features, use chi-square-based association measures instead of numeric correlation. section 2.2.2 section 2.4.3
After finding a highly related feature group, decide what the relationship means before removing columns:
- If one recorded feature causes another, retain the causative feature and drop the others; identifying causation requires domain knowledge, theoretical deductions, or causal analysis. section 2.2.2
- If features reflect an unrecorded latent factor, choose using domain knowledge plus target association and data quality, including errors, missingness, and outliers. section 2.2.2
- Remove exact duplicates immediately: they waste memory and computation without adding information. section 2.2.2
Study time and test scores
Time spent studying and test scores can be highly correlated because study time helps determine scores. In contrast, car performance and emissions may be associated because both are partly determined by fuel type, even when fuel characteristics are absent from the data. section 2.2.2
Listing 2.3 Finding multicollinear numeric features
import numpy as np
from sklearn.datasets import make_classification
from statsmodels.stats.outliers_influence import variance_inflation_factor
np.random.seed(0) #1
X, _ = make_classification(n_redundant=0,
n_repeated=0,
random_state=0) #2
X = np.hstack([X, X[:,:5] + np.random.random((X.shape[0],5))]) #3
vif = [variance_inflation_factor(X, i) for i in range(X.shape[1])] #4
print(np.round(vif,2))
for a in range(X.shape[1]): #5
for b in range(X.shape[1]):
if a < b:
r = np.corrcoef(X[:, a], X[:, b])[0][1] #6
if np.abs(r) > 0.90: #7
print(f"feature {a} and {b} have r={r:0.3f}")
This Python listing creates noisy copies of five features, then uses variance inflation factors and pairwise correlations to flag the resulting highly collinear pairs. section 2.2.2
