A language model predicts what token might come next, given the tokens seen so far. Its output is a probability distribution over possible next tokens. By repeatedly predicting a token and adding it to the sequence, it can generate longer text. section 15.1
Predicting Shakespeare-like text
A simple character-level language model can be trained to predict the next character in a sequence, then use repeated predictions to produce Shakespeare-like text. section 15.1
Predicting one token at a time makes generation manageable. With a 20,000-word vocabulary, a next-token model predicts among 20,000 possibilities; treating every four-word sequence as a separate output would require 160 quadrillion possibilities. section 15.1
A Transformer is a common architecture for processing text. Large foundation models used in generative AI are trained through self-supervised learning on massive amounts of unlabeled data. section 3.6.3 section 1.14







































































