A token is a discrete piece of text that an LLM converts into a number before processing. It may be a whole word, but often it is a subword such as “school” and “house” in “schoolhouse” or “thought” and “ful” in “thoughtful.” Tokens are the model’s basic input units, not letters or words in the human sense. section 2.1 section 2.2
Tokenization turns text into model-ready identifiers through four practical steps:
- Receive a text string from a user or another source. section 2.2.1
- Normalize the string when useful, such as changing uppercase letters to lowercase or removing irrelevant variation. section 2.2.1 section 2.2.2
- Segment the string into a sequence of discrete tokens, commonly using byte-pair encoding (BPE) to find frequent subword sequences. section 2.2.3
- Map every token to a unique integer identifier and collect those identifiers in the model’s vocabulary. section 2.2.1
Each identifier is initially just an abstract symbol; the model does not automatically know that related tokens share letters or meaning. During processing, tokens become embedding vectors, and their relationships are learned from patterns in training data. section 2.2 section 3.5
Why token boundaries matter
GPT-3 can represent “running” as one token while the slightly shorter “runnin” may require “run” and “nin.” A longer sentence therefore does not necessarily contain more tokens. Similarly, a number such as 123456 may be split into two tokens rather than six digit tokens, making arithmetic harder. Digit-level tokenization can improve arithmetic learning. section 2.2.4 section 2.3.2
Token choices create practical limits:
- Uncommon words and minor misspellings can receive very different token sequences; “Amoxicillin” and “Amoxicillan” share no common GPT-3 tokens. section 2.3.1
- Models that do not expose individual characters may struggle with word games, letter positions, and word length. section 2.3.1
- Languages represented with more granular token sequences can require more processing time and cost more in per-token services. section 2.3.3
- Tokenizer design influences which data an LLM represents effectively, so difficult application failures may require a different approach or added domain tokens. section 2.3 section 2.5
Figure 2.5 A simplified byte pair encoding algorithm for creating tokens: first, find the most frequent pair of characters “ng.” Next, replace all instances of “ng” with a placeholder token “T,” and add “ng” to the vocabulary. Repeat the process until no common byte pairs remain.
Follow the simplified BPE sequence from frequent character pairs to merged subword tokens and an expanding vocabulary. section 2.2.3
