A token is a word or partial word into which text is divided for computational processing. A language model converts tokens into a mathematical representation, processes them, and converts its output back into readable text. In machine translation, for example, “The cat wore red socks” becomes “El gato usó calcetines rojos.” section 1.2
Tokens can also serve as markers for tool use. A model trained with examples of API calls can learn to produce special tokens that signal when to use a tool, such as search, and include its result in the response context. section 8.11
