A tokenizer converts text into a sequence of integer token IDs. The practical pipeline has three stages: standardize input lightly, split it into tokens, and index those tokens through a vocabulary. The term often covers the entire process, while the tokenizer object specifically maps text to token indices. section 14.2
Build the idea from a short sentence:
- Character splitting produces one token per character, including spaces. section 14.2
- Word splitting keeps words and selected punctuation as separate tokens. section 14.2
- Vocabulary lookup maps known tokens to fixed integers and sends unknown tokens to a special [UNK] index. section 14.2
- Pretrained models require their matching tokenizer because different token IDs do not correspond to the learned token representations. section 15.4.2
From sentence to IDs
For “The quick brown fox jumped over the lazy dog.”, a word tokenizer emits words and punctuation, then vocabulary lookup produces [0, 2, 3, 4, 5, 6, 1, 0, 7, 8]. Here, “The” and “lazy” become [UNK] because they are absent from the demonstrated vocabulary. section 14.2
Listing 15.20 Loading the RoBERTa pretrained model with KerasHub
import keras_hub tokenizer = keras_hub.models.Tokenizer.from_preset("roberta_base_en") backbone = keras_hub.models.Backbone.from_preset("roberta_base_en")
Load a tokenizer together with the matching RoBERTa pretrained model through KerasHub’s from_preset() interface. section 15.4.2
