A tokenizer converts text into the token sequence expected by a language model. For Transformer encoders, it splits text into vocabulary-defined word pieces before encoding produces one vector per token. section 13.3.1
Figure 13.4 The Transformer encoder
Follow the Transformer encoder pipeline from tokenization through contextual vector representations. section 13.3.1
A tokenizer applies the model's predefined tokenization scheme:
- Splits text into word pieces represented in the model vocabulary. section 13.3.1
- Represents “It’s” as three pieces: “it”, an apostrophe, and “s”. section 13.3.1
- Adds special pieces such as [CLS] and [SEP] to mark sentence boundaries. section 13.3.1
- Produces token values that the Transformer accepts as input rather than raw words. section 14.3.1
Tokenization also shapes long inputs for training. A sliding window divides a context into overlapping token lists, while truncation limits each list and padding makes every tensor the same length. section 14.3.1
Question-answering tokenization
Load a tokenizer trained for roberta-base, tokenize each question and context pair, and create tensors with a maximum of 384 tokens. Consecutive windows overlap by 128 tokens; padding fills the final window when necessary. section 14.3.1
Listing 14.9 Tokenizing our training set
# This function adapted from:
# https://github.com/huggingface/notebooks/blob/master/examples/
#question_answering.ipynb
# Copyright 2001, Hugging Face. Apache 2.0 Licensed.
from datasets import load_from_disk
from transformers import RobertaTokenizerFast
file = "data/question-answering/question-answering-training-set"
datadict = datasets.load_from_disk(file) #1
tokenizer = from_pretrained("roberta-base") #2
...
def tokenize_dataset(examples):
maximum_tokens = 384 #3
document_overlap = 128 #4
pad_on_right = tokenizer.padding_side == "right" #5
tokenized_examples = tokenizer( #6
examples["question" if pad_on_right #6
else "context"], #6
examples["context" if pad_on_right #6
else "question"], #6
truncation="only_second" if pad_on_right #6
else "only_first", #6
max_length=maximum_tokens, #6
stride=document_overlap, #6
return_overflowing_tokens=True, #6
return_offsets_mapping=True, #6
padding="max_length" #6
)
... #7
return tokenized_examples
tokenized_datasets = datadict.map( #8
tokenize_dataset, #8
batched=True, #8
remove_columns=datadict["train"].column_names) #8
Inspect the training-set tokenization setup, including tokenizer loading, maximum length, overlap, padding direction, and paired inputs. section 14.3.1
