category_encoders is a Python package of scikit-learn-compatible encoders for categorical variables. Its TargetEncoder replaces each category with an expected target value: a mean for regression or a conditional probability or odds ratio for classification. section 6.1.3
Smoothing blends a category’s posterior target value with the dataset-wide prior, reducing overfitting when categories have few observations. The smoothing value must be positive and is selected experimentally, or replaced with a James Stein encoder. section 6.1.3
High-cardinality location features
A neighborhood feature with 221 values and a latitude–longitude grid feature with 2,259 values illustrate when target encoding can replace unwieldy one-hot representations. The encoder is configured on these columns and evaluated inside a five-fold pipeline with XGBoost. section 6.1.3
Listing 6.4 Using target encoding in the pipeline
from category_encoders.target_encoder import TargetEncoder
from XGBoost import XGBClassifier
from sklearn.model_selection import KFold, cross_validate
from sklearn.metrics import accuracy_score, make_scorer
target_encoder = TargetEncoder(cols=high_card_categorical, #1
smoothing=0.5) #2
accuracy = make_scorer(accuracy_score)
cv = KFold(5, shuffle=True, random_state=0)
xgb = XGBClassifier(booster='gbtree',
objective='reg:logistic',
n_estimators=300,
max_depth=4,
min_child_weight=3) #3
column_transform = ColumnTransformer(
[
('low_card_categories',
categorical_onehot_encoding,
low_card_categorical),
('high_card_categories',
target_encoder,
high_card_categorical),
('numeric',
numeric_standardization,
continuous)
],
remainder='drop',
verbose_feature_names_out=True,
sparse_threshold=0.0) #4
model_pipeline = Pipeline(
[('processing', column_transform),
('model', xgb)]) #5
cv_scores = cross_validate(estimator=model_pipeline,
X=data,
y=target_median,
scoring=accuracy,
cv=cv,
return_train_score=True,
return_estimator=True) #6
mean_cv = np.mean(cv_scores['test_score'])
std_cv = np.std(cv_scores['test_score'])
fit_time = np.mean(cv_scores['fit_time'])
score_time = np.mean(cv_scores['score_time'])
print(f"{mean_cv:0.3f} ({std_cv:0.3f})",
f"fit: {fit_time:0.2f}",
f"secs pred: {score_time:0.2f} secs") #7
Inspect the pipeline configuration for applying TargetEncoder to selected high-cardinality columns alongside one-hot encoding for lower-cardinality categories. section 6.1.3
