A tokenized text sequence is an ordered list of integer IDs, with each ID representing a token. Detokenization reverses the mapping and joins the resulting tokens into text. section 15.1
For text generation, predicting an entire sequence as one classifier output is impractical: a language model instead predicts the next token from the tokens seen so far. section 15.1
Why predict one token at a time?
With a vocabulary of 20,000 words, there are 20,000⁴ possible four-word sequences. Predicting one of 20,000 tokens at each step makes the prediction space more manageable. section 15.1
