A token is a discrete unit of text that a language model reads. Depending on the processing scheme, it may represent a word, punctuation mark, subword, or character. Tokens form the bridge between raw text and numerical representations: text is split into tokens, tokens become token IDs, and embedding layers retrieve vectors for model input. section 1.5 section 2.9
A practical token-processing flow is:
- Split text into words and special characters such as punctuation. section 2.2
- Use byte pair encoding to represent unfamiliar words as subword units or individual characters. section 2.9
- Convert tokens into integer token IDs, then use embedding lookup to produce continuous vectors. section 2.9
- Arrange tokenized data with a sliding window to create input-target pairs for training. section 2.0
Splitting a sentence into tokens
Applying a regular-expression tokenizer to "Hello, world. Is this-- a test?" produces ['Hello', ',', 'world', '.', 'Is', 'this', '--', 'a', 'test', '?']. The result preserves punctuation and separates the double hyphen as its own token. section 2.2
Figure 2.4 A view of the text processing steps in the context of an LLM. Here, we split an input text into individual tokens, which are either words or special characters, such as punctuation characters.
Shows how input text becomes individual word and punctuation tokens before embedding and model processing. section 2.2
