A token is a text unit a language model processes. It may be a whole word, a word fragment, or characters such as a trailing space. As a rule of thumb, one token represents about four characters of standard English text; this varies. section 5.3.1
Language models use tokens to process text and predict what comes next from the preceding text. They may choose a likely continuation or sometimes select a lower-probability one, so predictions are not always the single most likely option. section 5.3.1
An electric-vehicle prompt
Given “The benefits of electric vehicles are …”, a model might predict “plentiful,” “numerous,” or “economic.” These continuations illustrate prediction; tokens need not correspond to complete words. section 5.3.1
