A vocabulary is the set of distinct tokens a language model recognizes. Think of it as a lookup list: each token has a numerical ID, so text can become a sequence of numbers the model can process. Tokens may be words, parts of words, characters, or punctuation. section 11.4 section 11.9
A small vocabulary can be built from training text in a few steps:
- Split words into characters and add an end-of-word marker, </w>, to preserve word boundaries. section 11.9
- Count adjacent token pairs. If i followed by n is the most frequent pair, merge it into the token in. section 11.9
- Repeat the merges, then assign an ID to each distinct token. The resulting list can contain both single characters and pieces such as in, ing, and th. section 11.9
Table 11.6 The vocabulary based on the training data
|
ID |
Token |
|
ID |
Token |
|
ID |
Token |
| 0 |
a |
|
12 |
o |
|
24 |
th |
| 1 |
l |
|
13 |
v |
|
25 |
k |
| 2 |
ic |
|
14 |
er |
|
26 |
, |
| 3 |
e |
|
15 |
y |
|
27 |
: |
| 4 |
w |
|
16 |
ti |
|
28 |
c |
| 5 |
s |
|
17 |
re |
|
29 |
r |
| 6 |
b |
|
18 |
d |
|
30 |
p |
| 7 |
g |
|
19 |
f |
|
31 |
u |
| 8 |
in |
|
20 |
i |
|
32 |
" |
| 9 |
n |
|
21 |
h |
|
33 |
? |
| 10 |
ing |
|
22 |
ers |
|
|
|
| 11 |
t |
|
23 |
on |
|
|
|
The sample vocabulary pairs each token with an ID; it includes characters, punctuation, and merged pieces. section 11.9
The IDs are labels, not meanings. Before a Transformer processes an ID, an embedding matrix looks up a trainable vector for it. Vocabulary size determines how many token rows that matrix needs. At prediction time, the model produces a score for every vocabulary token and converts those scores into probabilities for the next token. section 11.14 section 11.29
Vocabulary size is therefore an architecture choice as well as a result of preparing the text. A toy example uses 34 tokens, while the cited range for usable LLMs runs from 50,000 to more than 250,000. The practical takeaway: first decide what pieces of text count as tokens, then keep their IDs consistent from encoding through prediction. section 11.11 section 11.4
