An autoregressive language model is often described as writing word by word, but its prediction unit is the token. A token can be a whole word or a smaller piece of one: common words may stay intact, while rarer words may be split. Before a Transformer processes text, it converts it into tokens and then numbers. section 5.1
[4032, 17, 9821]
The integer sequence illustrates the numeric form of tokenized text. The numbers are token IDs, not words the model reads directly. section 5.1
Tokens also define the prediction task. During training, the model predicts the next token at each position. A causal mask prevents a position from attending to later tokens, so those predictions cannot borrow their answers from the future. Attention lets a Transformer connect tokens across a sequence without processing them one at a time as a recurrent neural network does. section 5.1
A token is therefore both a unit of prediction and a unit for measuring training data. WebText2 contained roughly 23 billion tokens after tokenization. Scaling experiments found that adding training data improved performance, though with diminishing returns; token counts became part of the practical question of how much data a larger model needs. section 6.1.3
When a model is short of tokens
The 2022 Chinchilla analysis estimated an optimal ratio of roughly 20 training tokens per parameter for Transformers of that era. GPT-3’s 175 billion parameters and 300 billion training tokens amounted to about 1.7 tokens per parameter. The comparison helped explain why a smaller model trained on more data could outperform a larger, undertrained one. section 6.1.7
