Normalization transforms text into a more consistent representation before it is split into tokens. Common changes include lowercasing characters, removing punctuation, standardizing Unicode encodings, or removing unusual characters. In a tokenization pipeline, normalization sits between receiving a text string and segmenting it into tokens. section 2.2.1 section 2.6
Normalization can make text easier to represent, but each transformation changes what information remains available:
- Lowercasing “Hello” and “hello” maps both forms to one representation, reducing vocabulary size and ambiguity caused by accidental capitalization. section 2.2.2
- Removing punctuation can prevent forms such as “hello” and “hello,” from becoming separate tokens under simple whitespace splitting. section 2.2.2 section 2.2.3
- Standardizing Unicode can collapse visibly identical characters into one encoding and remove characters such as zero-width spaces. section 2.2.4 section 2.6
- Preserving distinctions can matter: “Bill” may be a name, while “bill” may refer to money or another meaning. section 2.2.2
A useful transformation with a cost
Suppose a tokenizer receives “LLMs,” “LLms,” and “llms.” Lowercasing and removing punctuation can turn these variations into one simpler form, shrinking the vocabulary. The same policy can erase meaningful distinctions, however: lowercasing “Bill” removes a clue that the word is probably a person’s name. Normalization therefore trades representational efficiency for potentially lost information. section 2.2.2
Figure 2.3 The normalization process commonly involves changing text to remove uppercase characters and punctuation.
Use the normalization diagram to trace how uppercase characters and punctuation are removed before text is segmented into tokens. section 2.2.2
Normalization also has a security and cost dimension. Homoglyphs and invisible Unicode characters can produce different byte encodings, inflate token counts, alter parsing, and increase computation. Services may replace look-alike characters with canonical forms, although modern tokenizers often perform little normalization beyond removing some nonprinting characters. section 2.2.4 section 2.2.3
