A character-based NLP pipeline represents text as individual letters, characters, or graphemes rather than relying only on whole words. Machines can detect patterns across character sequences that people may overlook, while character-level representations can accommodate large or unfamiliar vocabularies. section 2.2
Character-level processing is useful when token design must balance:
- Flexibility for prediction and text generation. section 2.2
- Human explainability, which is stronger with word tokenizers than with machine-efficient encodings. section 2.2
- Computational efficiency, which can favor byte pair, word piece, or sentence piece encoding over readable word tokens. section 2.2
Hidden meaning in a short command
The command “Don’t!” illustrates how small character sequences can participate in layered meaning: the visible command can imply expanded phrases such as “Don’t you do that!” and “You, do not do that!” section 2.2
