Token type IDs are integer labels that identify which input segment each token belongs to. Hugging Face tokenization adds them alongside input_ids and attention_mask in the processed dataset. section 5.2.2
from transformers import AutoTokenizer
dataset = load_dataset(dataset_id)
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
tokenized_dataset = dataset.map(
lambda examples:
tokenizer(examples['text'],
truncation = True,
padding = 'max_length'),
batched = True)
Run the tokenizer mapping to produce tokenized dataset fields, including token_type_ids, with truncation and fixed-length padding. section 5.2.2
Tokenized dataset output
After tokenization, each dataset split exposes token_type_ids as a feature alongside text, label, input_ids, and attention_mask.
