String similarity assigns a numeric score to two strings: higher values indicate greater similarity. It helps recognize that “Data Scientsit” and “Data Scientist” are probably the same category even when exact matching treats them as different. section 8.4
This matters when manually entered labels create noisy, high-cardinality features. Similarity-based encoding gives related categories similar representations so downstream learning algorithms treat them alike. The challenge is computational: strings can differ in length, and matching subsequences across locations can become expensive. section 8.4
Practical uses include:
- Cleaning or merging misspelled and inconsistent categories. section 8.4
- Supporting autocorrect, database retrieval, and language translation. section 8.4
- Encoding categorical features with SimilarityEncoder, MinHashEncoder, or GapEncoder. section 8.4
Employee salary prediction
A salary dataset contains 385 employee-position-title categories. An ensemble compares SimilarityEncoder, MinHashEncoder, and GapEncoder alongside other preprocessing, achieving reported R² scores of 0.8996, 0.8997, and 0.8895 respectively. section 8.4
Listing 8.12 Encoding and ensembling with high-cardinality features
from sklearn.preprocessing import OneHotEncoder, MinMaxScaler
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from dirty_cat import SimilarityEncoder, MinHashEncoder, GapEncoder
from xgboost import XGBRegressor
from sklearn.metrics import r2_score
lo_card = ['gender', 'department_name',
'assignment_category'] #1
hi_card = ['employee_position_title'] #2
continuous = ['year_first_hired'] #3
encoders = [SimilarityEncoder,
MinHashEncoder(n_components=100),
GapEncoder(n_components=100)]
from sklearn.model_selection import train_test_split
Xtrn, Xtst, ytrn, ytst = \
train_test_split(X, y, test_size=0.2) #4
for encoder in encoders:
ensemble = XGBRegressor( #5
objective='reg:squarederror',learning_rate=0.1,
n_estimators=100, max_depth=3)
preprocess = ColumnTransformer(transformers=[
('continuous',
MinMaxScaler(), continuous), #6
('onehot',
OneHotEncoder(sparse=False), lo_card), #7
('dirty',
encoder, hi_card)], #8
remainder='drop')
pipe = Pipeline(steps=[('preprocess', preprocess),
('train', ensemble)]) #9
pipe.fit(Xtrn, ytrn)
ypred = pipe.predict(Xtst)
print('{0}: {1}'.format(encoder.__class__.__name__,
r2_score(ytst, ypred))) #10
Inspect the complete preprocessing and ensemble pipeline for high-cardinality string features. section 8.4
