Tokenization segments text into units such as words, subwords, or punctuation. In search and language-model pipelines, these units become inputs for phrase extraction, spelling analysis, semantic encoding, or model training. section 6.4.2 section 14.3.1
Choose tokenization behavior according to the task:
- Split queries with a regular-expression tokenizer, then remove stop words, short terms, and numeric-only terms when counting useful keywords. section 6.5.2
- Keep a query whole when multiword spelling variations matter; tokenizing separately can lose product-name context. section 6.5.2
- For question answering, convert questions and contexts into model vocabulary tokens, then split long contexts into overlapping windows and pad them to equal length. section 14.3.1
Tokenizing query signals
A query such as “hp touchpad 32” can be treated as separate terms for keyword-frequency analysis or as one unit for multiterm spelling correction. Whole-query processing can map variants such as “iphone4 s” and “iphones 4s” to “iphone 4s”, but may also lose relevant context when shortening a query. section 6.5.2
Listing 14.9 Tokenizing our training set
# This function adapted from:
# https://github.com/huggingface/notebooks/blob/master/examples/
#question_answering.ipynb
# Copyright 2001, Hugging Face. Apache 2.0 Licensed.
from datasets import load_from_disk
from transformers import RobertaTokenizerFast
file = "data/question-answering/question-answering-training-set"
datadict = datasets.load_from_disk(file) #1
tokenizer = from_pretrained("roberta-base") #2
...
def tokenize_dataset(examples):
maximum_tokens = 384 #3
document_overlap = 128 #4
pad_on_right = tokenizer.padding_side == "right" #5
tokenized_examples = tokenizer( #6
examples["question" if pad_on_right #6
else "context"], #6
examples["context" if pad_on_right #6
else "question"], #6
truncation="only_second" if pad_on_right #6
else "only_first", #6
max_length=maximum_tokens, #6
stride=document_overlap, #6
return_overflowing_tokens=True, #6
return_offsets_mapping=True, #6
padding="max_length" #6
)
... #7
return tokenized_examples
tokenized_datasets = datadict.map( #8
tokenize_dataset, #8
batched=True, #8
remove_columns=datadict["train"].column_names) #8
Shows a RoBERTa tokenizer preparing question-answering data with 384-token windows, 128-token overlap, truncation, and padding. section 14.3.1
