A feature interaction occurs when a prediction depends on the joint values of two or more inputs, not on each input independently. For a linear regression model, create an interaction as a new feature by multiplying features together. This can make a relationship with the target more nearly linear and can capture domain knowledge that individual columns miss. section 4.3.1 section 4.3.2
Interactions are especially useful in tabular business problems where a combination changes the meaning of a value. A large number of bedrooms may suggest a hostel, while a high minimum-stay requirement may indicate seasonal accommodation; a simple linear baseline can miss such nuances and produce implausible predictions. section 7.2.3
Choose an approach based on the model and the data:
- For linear models, engineer selected products of features when domain knowledge suggests a joint effect. Keep the resulting feature set meaningful and manageable. section 4.3.1
- For broad automated coverage, use PolynomialFeatures to generate products and powers up to a chosen degree. This can model joint effects and curves, but greatly expands the feature set. section 4.3.2
- Use regularization after polynomial expansion to control the extra features; expansion can add noise, multicollinearity, and overfitting risk. section 4.3.2
- Let decision trees and tree-based ensembles learn interactions through successive splits, but precompute known important interactions for GBDT when doing so can reduce required splits. section 6.1.4
Combining advertising categories before encoding
Suppose website and advertising format are both high-cardinality categorical features. Target-encoding each separately can prevent a later model from capturing their joint effect. Instead, first create one categorical feature that combines website and format, then target-encode that combination. section 6.1.3
Listing 6.8 Discovering interactions by XGBFIR
import xgbfir
xgbfir.saveXgbFI(
model_pipeline['XGBoost'],
feature_names=(
model_pipeline['processing']
.get_feature_names_out()
),
OutputXlsxFile='fir.xlsx') #1
fir = pd.read_excel('fir.xlsx', sheet_name='Interaction Depth 1') #2
result = fir[["Interaction", "Gain"]].sort_values(by="Gain",
ascending=False).head(10).round(2) #3
for index, row in result.iterrows():
print(f"{row['Interaction']}")
PartialDependenceDisplay.from_estimator(
model_pipeline,
X=data,
kind='average',
features=[(
'minimum_nights',
'calculated_host_listings_count')]) #4
Use XGBFIR to rank interactions learned by an XGBoost model by gain, then inspect a promising pair with a conjoint partial dependence plot. section 6.1.4
Interactions are not automatically better. More engineered products can improve predictive power, yet also make models noisier, less stable under multicollinearity, and more prone to memorizing training examples. Prefer a small set of justified interactions or use regularization when systematic expansion is necessary. section 4.3.2 section 4.3.1
