Normalization standardizes text so equivalent content uses consistent characters, casing, whitespace, and related formatting. This helps prevent one meaning from being split across unnecessary tokens. section 11.7
Common normalization operations include:
- Lowercase words to treat forms such as “The” and “the” as one token, reducing vocabulary size. Modern language-model pipelines may preserve casing when it carries meaning, such as “apple” versus “Apple.” section 11.7
- Convert smart quotes to straight quotes so different character representations become consistent. section 11.7
- Collapse runs of spaces, tabs, and line breaks into a single space. section 11.7
- Choose how to handle accented characters and other special characters consistently. section 11.7
A small text cleanup
The forms “The cafe” and “the cafe” can be normalized by converting smart quotes, collapsing extra whitespace, and lowercasing. The result is more uniform text, although casing should be preserved when distinctions such as “apple” and “Apple” matter. section 11.7
Figure 11.12 Training data normalized
Use the normalized training-data figure to see the cleanup process as a visible before-and-after transformation. section 11.7
