A token is a basic unit of text processed by a system. In BLEU’s original formulation, tokens are typically words separated by whitespace and punctuation. Modern LLMs may split rare words into subword pieces; for BLEU and ROUGE, treating tokens as words works well. section 2.1.1
Word and subword units
The word “unhappiness” might be split into subword tokens such as “un”, “h”, “happiness”, and “ness.” section 2.1.1







































