Tokenization splits raw text into words, subwords, and punctuation tokens that an LLM can encode as vectors before transformer processing. It preserves capitalization and may retain or remove whitespace depending on the application. section 2.0 section 2.2
A practical tokenization flow proceeds as follows:
- Read a text sample and split it initially on whitespace. section 2.2
- Extend the split pattern to separate punctuation and multi-character symbols such as --. section 2.2
- Remove redundant whitespace tokens when exact spacing is not required. section 2.2
- Pass the resulting tokens to token encoding, vector embeddings, and input-output sampling for LLM training. section 2.0 section 2.1
Separating words and punctuation
For `Hello, world. Is this-- a test?`, a regular-expression tokenizer produces `['Hello', ',', 'world', '.', 'Is', 'this', '--', 'a', 'test', '?']`. The same approach can preprocess a larger text sample while retaining capitalization.
