A dimension is one numerical coordinate used to describe something. For an embedding, the dimension is the number of floating-point values describing each token; wider vectors can hold more meaning-related detail. section 11.11 section 7.2
The same idea appears in several practical settings:
- In optimization, each measured property adds a dimension: pepper size and color create three-dimensional data, while location, age, and fertilizer can produce six dimensions. section 7.2
- In a dataset, dimensions can be physical measurements such as a diamond’s X, Y, and Z length, width, and depth. section 8.7
- In a Transformer, embedding width sets how many floating-point numbers represent one token at each layer. section 11.11
From two dimensions to an embedding
Imagine describing peppers by size and spiciness: each pepper becomes a point on a two-dimensional chart. Add color and the points need three coordinates; add location, age, and fertilizer and the search space reaches six dimensions. A token embedding applies the same pattern, storing each token as a vector with a chosen number of numerical coordinates. section 7.2 section 11.11
Table 11.7 The hyperparameters for the Transformer framework
|
Hyperparameter |
Rule of thumb |
Toy LLM example |
Real-world LLM |
| Vocabulary size (V) |
Based on training data |
34 sentence pieces |
50,000 (more than 250,000 sentence pieces |
| Context window (T) |
Longest prompt you need, rounded to power of 2, that still fits batch on a GPU |
32 tokens |
1,024 (more than 32,000 tokens) |
| Width (d) |
Start near 4√Vocabulary size |
4 channels |
768 (more than 20,000 channels) |
| Depth (L) |
1–2 layers per 100,000 characters of training text |
2 layers |
12 (more than 120 layers |
| Attention heads (h) |
h = Width ÷ 64 (round to even) |
1 head |
12 (more than 128 heads |
| Feed-forward size (dn) |
dn = 4 × Width is a safe default |
Layer 1: 8 neurons Layer 2: 4 neurons |
3,027 (more than 80,000 neurons) |
Compare toy and real-world Transformer embedding widths and the rule of thumb for choosing width. section 11.11
