A tokenizer converts raw text into model-specific tokens and numerical IDs that models consume during training and inference. Tokens may represent words, subwords, characters, punctuation, or special characters. section 2.1.1 section 2.6
Operationally, tokenization determines how text fits and flows through a model:
- Match the tokenizer to the model architecture, because pretrained models expect specific token formats. section 2.1.1
- Use padding and truncation to create consistent-length inputs within the model’s maximum token limit. section 2.1.1
- Handle architecture-specific markers: BERT uses [CLS] and [SEP], while GPT workflows tokenize context for text generation. section 2.1.1 section 2.6
- Account for token count when estimating processing time, memory use, and available prompt capacity. section 2.1.1
BERT classification input
Load BertTokenizer with bert-base-uncased, format text with [CLS] and [SEP], then request padded, truncated PyTorch tensors for model input.
Question-answering inference
Tokenize a question together with its context, pass the tensors to the model, select predicted start and end positions, and decode the answer tokens.
