Generation samples new text from a model’s learned distribution. A language model can produce sentences, paragraphs, or multiturn conversations; its learned probabilities reflect both grammatical form and meaning. section 5.1
Most modern large language models generate autoregressively: they predict one word or token at a time, conditioning each prediction on what came before. The probability of a sentence becomes a chain of conditional probabilities. During training, a causal mask blocks access to future tokens, while shifted sequences provide next-token labels at every position. section 5.1
Grammar and meaning shape probability
“The mouse ate the cheese” is plausible; “The the mouse ate cheese” is grammatically broken, while “The cheese ate the mouse” is grammatical but semantically odd. A capable language model assigns these examples different probabilities, reflecting both syntax and semantics. section 5.1
