A model needs numbers, not raw text. Tokenization turns a string into integer data in three steps: standardize the text, split it into tokens, and index each token with a vocabulary. The split might produce characters, words, or subwords; that choice changes both sequence length and vocabulary size. section 14.6 section 14.2
Split a sentence
For “The quick brown fox jumped over the dog.”, the pattern `[\w]+|[.,!?;]` produces `The`, `quick`, `brown`, `fox`, `jumped`, `over`, `the`, `dog`, `.`. Punctuation becomes a token rather than disappearing. A character split instead yields individual letters and spaces. Neither split is numeric yet: vocabulary lookup supplies the integer IDs. section 14.2
A simple tokenizer can lowercase its input, split it, then look up each token’s ID. Build the vocabulary from token counts and reserve `[UNK]` for anything it does not contain. Capping vocabulary size limits model parameters, but sends more uncommon tokens to `[UNK]`. Apply the same standardization and split rule when building the vocabulary and tokenizing new text. section 14.2.1
Listing 14.3 A basic word-level tokenizer
class WordTokenizer: def __init__(self, vocabulary): self.vocabulary = vocabulary self.unk_id = vocabulary["[UNK]"] def standardize(self, inputs): return inputs.lower() def split(self, inputs): return re.findall(r"[\w]+|[.,!?;]", inputs) def index(self, tokens): return [self.vocabulary.get(t, self.unk_id) for t in tokens] def __call__(self, inputs): inputs = self.standardize(inputs) tokens = self.split(inputs) indices = self.index(tokens) return indices
The word-level implementation keeps standardization, splitting, and indexing separate, making the effect of each step easy to inspect. section 14.2.1
The trade-off is concrete: a character tokenizer covers the full tested novel with 64 vocabulary entries but produces long sequences. A word tokenizer produces much shorter sequences, yet fills a 2,000-entry vocabulary; indexing every word in that novel would require about 17,000 entries. Subword tokenization is a third option and is usually the most effective with a sufficiently expressive model and enough training data. section 14.2.1 section 14.6
Match the output to the model. For a bag-of-words classifier, Keras `TextVectorization` can learn a vocabulary with `adapt()` and turn word tokens into a multi-hot vector. For fine-tuning RoBERTa on movie reviews, use its tokenizer and pack IDs with the start, end, and padding tokens expected from pretraining; `StartEndPacker` can also trim long sequences and return a padding mask. As a rule of thumb, do not change a pretrained model’s token arrangement casually. section 14.4.1 section 15.4.3
Listing 15.21 Preprocessing IMDb movie reviews with RoBERTa’s tokenizer
def preprocess(text, label): packer = keras_hub.layers.StartEndPacker( sequence_length=512, start_value=tokenizer.start_token_id, end_value=tokenizer.end_token_id, pad_value=tokenizer.pad_token_id, return_padding_mask=True, ) token_ids, padding_mask = packer(tokenizer(text)) return {"token_ids": token_ids, "padding_mask": padding_mask}, label preprocessed_train_ds = train_ds.map(preprocess) preprocessed_val_ds = val_ds.map(preprocess) preprocessed_test_ds = test_ds.map(preprocess)
The preprocessing function shows tokenization followed by sequence packing, producing token IDs and a padding mask for movie reviews. section 15.4.3
