Tokenization divides text into model-processing units called tokens. A token may be a word, subword, character, punctuation mark, or other linguistic element, depending on the selected strategy. section 3.1.1
Tokenize text with BERT
from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") input_text = "What is unhappiness?" tokens = tokenizer.tokenize(input_text) print(tokens) # ['what', 'is', 'un', '##ha', '##pp', '##iness', '?'] section 3.1.1
Choose a strategy according to the language and task:
- Word-level tokenization splits text into words using whitespace or punctuation. It is straightforward but struggles with out-of-vocabulary words and does not represent internal word structure. section 3.1.1 section 5.2.1
- Subword-level tokenization splits words into meaningful fragments such as prefixes, roots, and suffixes. It handles rare, misspelled, compound, and morphologically complex words more flexibly. section 3.1.1 section 5.2.1
- Character-level tokenization represents individual letters, digits, and punctuation. It suits languages with less obvious word boundaries but can enlarge vocabularies and reduce higher-level interpretability. section 3.1.1 section 5.2.1
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") #1
input_text = "What is unhappiness?"
tokens = tokenizer.tokenize(input_text, return_tensors="pt") #2
print(f"{tokens = }")
Inspect the Hugging Face tokenizer call that produces subword tokens, then compare its output with the displayed token list. section 3.1.1
For model inference, the tokenizer converts text into numerical fields such as input_ids and attention_mask. input_ids identify tokens; attention_mask distinguishes actual tokens from padding. token_type_ids can distinguish multiple input segments, as in BERT. section 3.2.3 section 5.2.2
Tokenize a dataset for training
from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased") def tokenize_function(examples): return tokenizer( examples["text"], padding="max_length", truncation=True, max_length=512 ) tokenized_datasets = dataset.map(tokenize_function, batched=True) section 6.1.3
from transformers import AutoTokenizer
dataset = load_dataset(dataset_id)
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
tokenized_dataset = dataset.map(
lambda examples:
tokenizer(examples['text'],
truncation = True,
padding = 'max_length'),
batched = True)
Follow the dataset mapping pattern for batched tokenization, truncation, and max-length padding. section 5.2.2
