Text standardization normalizes input before tokenization so accidental encoding differences do not become separate tokens. Lowercasing, punctuation removal, and sometimes accent removal make nearly identical sentences easier to represent consistently. section 14.2
Making two sentences more alike
“sunset came. i was staring at the Mexico sky. Isnt nature splendid??” and “Sunset came; I stared at the México sky. Isn’t nature splendid?” initially produce different representations. Lowercasing and removing punctuation yields nearly identical text; removing accents can reduce the remaining difference. section 14.2
A practical standardization pipeline can use these decisions:
- Convert input to lowercase when capitalization should not distinguish tokens. section 14.2.1
- Remove punctuation when punctuation is not useful for the modeling task. section 14.2
- Remove accent marks when accented and unaccented forms should be treated alike. section 14.2
- Apply the same choices while building the vocabulary and transforming new inputs. section 14.2.1
Standardization is the first stage of tokenization: it normalizes text, splitting creates tokens, and indexing assigns integer IDs. It is useful feature engineering, but it should be chosen deliberately because some tasks need case or punctuation. section 14.6 section 15.1.1
Figure 14.1 The text preprocessing pipeline
Use the preprocessing pipeline to place standardization before splitting and indexing. section 14.2
