Tokenization divides natural-language text into discrete units so computers can count, compare, search, classify, and represent documents numerically. A tokenizer turns an unstructured string into data suitable for an NLP pipeline. section 2.0
Tokenization can operate at several granularities:
- Words, punctuation, emojis, emoticons, and logograms provide familiar, interpretable tokens. section 2.1
- Word pieces, characters, and graphemes capture smaller units that can transfer patterns across related or unfamiliar words. section 2.2
- Byte pair encoding, word-piece encoding, and sentence-piece encoding improve computational efficiency while making tokens less understandable to humans. section 2.2
Reducing false greeting matches
A tokenized intent recognizer can detect Hi or Hello without mistaking Hiking for Hi or Hell for Hello. This reduces false positives, although covering slang, misspellings, and typos still requires broader handling. section 2.5
Token counts support keyword detection, full-text search, information retrieval, sentiment classification, and document comparison. Token sequences can also become word vectors or one-hot encoded matrices, while normalization, stemming, and stop-word removal trade precision against recall and information loss. section 2.0 section 2.5 section 2.8
