When strings are long or vary in wording, compare their tokenized parts rather than requiring identical character positions. Tokens are typically words or n-grams—substrings of a chosen length—and the resulting similarity score indicates how closely two strings match. section 8.4
A practical workflow is:
- Decompose each longer string into tokens such as words or n-grams. section 8.4
- Measure overlap or similarity between the token collections, producing a numeric score. section 8.4
- Use the scores to group similar high-cardinality categories or reduce the impact of typos in dirty categories. section 8.4
- Prefer character-based similarity for short strings; token-based methods are aimed at longer strings. section 8.4
Listing 8.12 Encoding and ensembling with high-cardinality features
from sklearn.preprocessing import OneHotEncoder, MinMaxScaler
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from dirty_cat import SimilarityEncoder, MinHashEncoder, GapEncoder
from xgboost import XGBRegressor
from sklearn.metrics import r2_score
lo_card = ['gender', 'department_name',
'assignment_category'] #1
hi_card = ['employee_position_title'] #2
continuous = ['year_first_hired'] #3
encoders = [SimilarityEncoder,
MinHashEncoder(n_components=100),
GapEncoder(n_components=100)]
from sklearn.model_selection import train_test_split
Xtrn, Xtst, ytrn, ytst = \
train_test_split(X, y, test_size=0.2) #4
for encoder in encoders:
ensemble = XGBRegressor( #5
objective='reg:squarederror',learning_rate=0.1,
n_estimators=100, max_depth=3)
preprocess = ColumnTransformer(transformers=[
('continuous',
MinMaxScaler(), continuous), #6
('onehot',
OneHotEncoder(sparse=False), lo_card), #7
('dirty',
encoder, hi_card)], #8
remainder='drop')
pipe = Pipeline(steps=[('preprocess', preprocess),
('train', ensemble)]) #9
pipe.fit(Xtrn, ytrn)
ypred = pipe.predict(Xtst)
print('{0}: {1}'.format(encoder.__class__.__name__,
r2_score(ytst, ypred))) #10
A salary-prediction pipeline compares SimilarityEncoder, MinHashEncoder, and GapEncoder while separating low-cardinality, high-cardinality, and continuous features. section 8.4
Employee salary categories
Employee position titles contain many distinct categories, making them a candidate for similarity-based encoding. In a held-out salary experiment, SimilarityEncoder, MinHashEncoder, and GapEncoder achieved R² values of 0.8996, 0.8997, and 0.8895 respectively. section 8.4
