AutoTokenizer is a Transformers factory class that selects and loads the tokenizer associated with a specified pretrained model. Pass a model identifier to from_pretrained(); the tokenizer can then convert text into model-ready token IDs and masks. section 3.2.3 section 5.2.2
Load a tokenizer for sentiment analysis
from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained( "distilbert/distilbert-base-uncased-finetuned-sst-2-english" ) inputs = tokenizer("I loved the movie, it was fantastic!", return_tensors="pt") The result is a dictionary containing input_ids and attention_mask tensors that can be passed to a PyTorch model. section 3.2.3
Use the tokenizer to:
- Load the tokenizer matching a pretrained model identifier. section 5.2.2
- Convert text into input_ids and attention_mask values, with optional token_type_ids for some models. section 3.2.3 section 5.2.2
- Tokenize dataset rows efficiently with dataset.map(), using batching, truncation, and max-length padding. section 5.2.2
- Inspect token IDs, convert IDs back to tokens, and identify padding through the attention mask. section 5.2.2
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"distilbert/distilbert-base-uncased-finetuned-sst-2-english")
Browse the minimal loading example for a tokenizer tied to a pretrained sentiment-analysis model. section 3.2.3
