liveBook · Manning

Language model

As explained in Deep Learning with R, Third Edition

A large language model is a machine-learning model trained on large text corpora to understand and generate natural-language text.

A language model predicts what token might come next, given the tokens seen so far. Its output is a probability distribution over possible next tokens. By repeatedly predicting a token and adding it to the sequence, it can generate longer text. section 15.1

Predicting Shakespeare-like text

A simple character-level language model can be trained to predict the next character in a sequence, then use repeated predictions to produce Shakespeare-like text. section 15.1

Predicting one token at a time makes generation manageable. With a 20,000-word vocabulary, a next-token model predicts among 20,000 possibilities; treating every four-word sequence as a separate output would require 160 quadrillion possibilities. section 15.1

A Transformer is a common architecture for processing text. Large foundation models used in generative AI are trained through self-supervised learning on massive amounts of unlabeled data. section 3.6.3 section 1.14

How Deep Learning with R, Third Edition covers it

How deep it goes
Defines the conditional next-token distribution and motivates it against whole-sequence classification.
Examples
Uses character prediction for Shakespeare-like text and a 20,000-word vocabulary to illustrate sequence complexity.
What you can do after
Evaluate and understand why predicting one token at a time makes text generation tractable.
How it connects
Connects next-token prediction with text generation and Transformer architecture.

Also explained in

Compare how 72 books explain language model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.