A token is a unit of text a language model processes: a whole word, part of a word, or punctuation. Tokenizers vary, but a rough estimate is four tokens per three words. section 1.1
Tokenization does more than split text at spaces. For example, a tokenizer may split “don’t” into “do” and “n’t”; subword schemes can represent common words whole and break less common words into smaller pieces. Each model has its own tokenization, so tokens prepared for one model may be misleading to another. section 1.3.2
A model’s context window covers both prompt and response tokens. When the input exceeds it, early tokens can be dropped without warning, leaving the model to respond without important context. Summarizing key findings can help keep them in view. section 1.1 section 1.3.2
