A language model’s vocabulary is its fixed set of tokens: common words, word pieces, characters, UTF-8 symbols, and control tokens it can read and generate. Tokens let the model handle unfamiliar words by combining smaller pieces instead of requiring every complete word. section 1.8
English versus French
English text often maps to useful whole-word tokens, while French may be split into many smaller fragments. A French translation of the U.S. Constitution’s preamble takes more than twice as many tokens as the English version, with pieces such as “ats” and “-Un” carrying little meaning alone. More tokens can increase cost and make contextual interpretation harder. section 1.11
The vocabulary balances coverage against size:
- A larger vocabulary could store more complete words and derivatives, but would waste capacity on closely related meanings and make the model less efficient. section 1.12
- Common words receive compact tokens, while uncommon words and non-Latin text may require several characters or UTF-8 pieces. section 1.11 section 1.8
We| the| People| of| the| United| States|,| in| Order| to| form| a| more| perfect| Union|,| establish| Justice|,| insure| domestic| Tran|qu|ility|,| provide| for| the| common| defense|,| promote| the| general| Welfare|,| and| secure| the| Bless|ings| of| Liberty| to| ourselves| and| our| Poster|ity|,| do| ord|ain| and| establish| this| Constitution| for| the| United| States| of| America|.
Inspect the Constitution’s English tokenization, including whole words, punctuation, spaces, and fragments. section 1.11
Nous|,| le| Pe|uple| des| É|t|ats|-Un|is|,| en| vue| de| former| une| Union| plus| par|fa|ite|,| d|'é |tabl|ir| la| justice|,| de| faire| rég|ner| la| pa|ix| int|érie|ure|,| de| pour|voir| à| la| déf|ense| commune|,| de| dévelop|per| le| bien|-être| général| et| d|' |ass|urer| les| bien|fa|its| de| la| libert|é| à| nous|-m|ê|mes| et| à| notre| post|é|rit|é|,| nous| dé|cr|é|tons| et| é|tab|lis|sons| cette| Constitution| pour| les| É|t|ats|-Un|is| d|' |Am|érique|.
Compare the French tokenization with the English listing to see how vocabulary choices produce longer, less meaningful fragments. section 1.11
