A language model learns patterns in text so it can predict what comes next. Think of a phone keyboard suggesting “you,” “we,” or “the” after “Hey, what are.” A simple version can use a table of learned probabilities. A large language model (LLM) uses the same basic idea, but learns far more patterns from far more text. section 11.1
An LLM is a trained machine learning model for natural language processing. Its many adjustable parameters capture patterns involving words, grammar, facts, and writing styles. Training data may include web pages, novels, research papers, forum posts, and code. Unlike a rulebook written by programmers, the resulting model is an artifact built by learning from data. section 11.1 section 1.5
Three upgrades help turn autocomplete into a more capable language model:
- Capacity: many parameters hold more subtle language patterns than a small probability table. section 11.1
- Data: a much larger collection of text gives the model more examples to learn from. section 11.1
- Architecture: modern LLMs commonly use Transformers. Attention selects earlier words to focus on; feed-forward neural networks process information; normalization layers are also part of a Transformer block. section 11.1
From a suggestion to a response
Type “Hey, what are.” A keyboard might suggest one likely next word using its probability table. An LLM can use context to generate a longer, coherent continuation. That expanded capability also supports answering questions, translating languages, and writing code. Attention is like highlighting earlier words that matter while deciding what comes next. section 11.1
Size is a practical choice, not a free upgrade. More complex architectures can capture more nuance, but need more memory and computing power to train and may take longer to answer. A small learning example uses a vocabulary of 34 tokens; a usable LLM may use 50,000 to more than 250,000. Architecture choices therefore depend on the problem and the available resources. section 11.11
Table 11.7 The hyperparameters for the Transformer framework
|
Hyperparameter |
Rule of thumb |
Toy LLM example |
Real-world LLM |
| Vocabulary size (V) |
Based on training data |
34 sentence pieces |
50,000 (more than 250,000 sentence pieces |
| Context window (T) |
Longest prompt you need, rounded to power of 2, that still fits batch on a GPU |
32 tokens |
1,024 (more than 32,000 tokens) |
| Width (d) |
Start near 4√Vocabulary size |
4 channels |
768 (more than 20,000 channels) |
| Depth (L) |
1–2 layers per 100,000 characters of training text |
2 layers |
12 (more than 120 layers |
| Attention heads (h) |
h = Width ÷ 64 (round to even) |
1 head |
12 (more than 128 heads |
| Feed-forward size (dn) |
dn = 4 × Width is a safe default |
Layer 1: 8 neurons Layer 2: 4 neurons |
3,027 (more than 80,000 neurons) |
Compare the toy Transformer’s vocabulary, context window, layers, and attention heads with the larger configurations shown for a real-world LLM. section 11.11
