A language model assigns probabilities to text. A sentence that is grammatical and plausible receives a higher probability than one with broken grammar or an unlikely meaning. section 5.1
Syntax and meaning
“The mouse ate the cheese” is grammatical and plausible. “The the mouse ate cheese” breaks grammar; “The cheese ate the mouse” is grammatical but semantically odd. A language model can assign both unusual sentences lower probabilities. section 5.1
Modern generative language models can sample from learned text distributions to produce new text or participate in conversations. Most are autoregressive: they predict each next token from what came before. Training uses shifted sequences as labels, while a causal mask prevents predictions from using future tokens. section 5.1
