Redundant feature

A redundant feature duplicates, nearly duplicates, or conveys substantially the same information as another feature. In tabular data, redundancy includes identically copied columns under different names, highly collinear numeric columns, similarly encoded categorical columns, and numeric and categorical representations derived from one source. Removing exact duplicates saves memory and computation without losing information. section 2.2.2 section 2.4.3

Treat redundancy as a data-quality problem to investigate before modeling. It can make learning more complicated and expensive; alongside noise, flipped labels, and irrelevant variables, redundant and collinear features made the Madelon prediction task difficult. Redundant characteristics can also challenge neural networks on tabular data. section 2.2 section 1.4

Distinguish exact duplication from strong association before deciding what to keep:

• Discard duplicated columns immediately: they add no information and consume memory and compute. section 2.2.2
• For highly collinear numeric features, inspect pairwise correlations. Correlations near 1 or -1 indicate collinearity; a practical screening range is an absolute correlation above 0.98 to 0.99. section 2.4.3
• For strongly associated categorical features, use chi-square-based association measures such as Cramér’s V. section 2.2.2 section 2.4.3
• Use domain knowledge, causal analysis, or further modeling to choose between related features. If one feature causes another, retain the causative feature; otherwise compare target association and data quality, including errors, missing values, and outliers. section 2.2.2

import numpy as np
from sklearn.datasets import make_classification
from statsmodels.stats.outliers_influence import variance_inflation_factor
 
np.random.seed(0)                                                   #1
X, _ = make_classification(n_redundant=0,
                           n_repeated=0,
                           random_state=0)                          #2
X = np.hstack([X, X[:,:5] + np.random.random((X.shape[0],5))])      #3
 
vif = [variance_inflation_factor(X, i) for i in range(X.shape[1])]  #4
print(np.round(vif,2))
 
for a in range(X.shape[1]):                                         #5
    for b in range(X.shape[1]):
        if a < b:
            r = np.corrcoef(X[:, a], X[:, b])[0][1]                 #6
            if np.abs(r) > 0.90:                                    #7
                print(f"feature {a} and {b} have r={r:0.3f}")

This listing constructs numeric features with near-duplicates, then uses variance inflation factors and pairwise correlations to flag multicollinearity. section 2.2.2

A correlation matrix heatmap is readable for a small set of numeric features. With many columns, list only correlations above a chosen threshold instead. Correlation reveals association, not which feature is causative; selecting among related features still requires contextual judgment. section 2.4.3 section 2.2.2