A token is a chunk of text that a machine learning model processes as one input unit. Depending on the tokenizer, it may represent a word, punctuation mark, subword, character, or other linguistic element. section 3.1.1 section 7.1
Inspect BERT tokens with Hugging Face
from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") input_text = "What is unhappiness?" tokens = tokenizer.tokenize(input_text, return_tensors="pt") print(f"{tokens = }") # ['what', 'is', 'un', '##ha', '##pp', '##iness', '?'] section 3.1.1
Common tokenization strategies include:
- Word tokenization splits text into words using whitespace or punctuation. section 3.1.1
- Subword tokenization splits words into units such as prefixes, suffixes, or roots, supporting varied vocabularies and linguistic forms. section 3.1.1
- Character-level tokenization splits text into individual letters, digits, and punctuation marks, but can reduce higher-level interpretability. section 3.1.1
- Tokenization can also convert text into numerical IDs required by most NLP models. section 5.2
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") #1
input_text = "What is unhappiness?"
tokens = tokenizer.tokenize(input_text, return_tensors="pt") #2
print(f"{tokens = }")
Use the listing to inspect how a pretrained BERT tokenizer converts an input sentence into model-ready token units. section 3.1.1
