A tokenizer splits text into model-ready tokens, then maps those tokens to discrete indexes. A pretrained subword tokenizer handles multilingual text more efficiently than a custom word-level tokenizer. section 10.1.1
A practical tokenization pipeline:
- Load a pretrained XLM tokenizer through Hugging Face Transformers. section 10.1.1
- Split sentences into complete words, punctuation, or subwords such as “fr” and “ench,” using </w> to mark word boundaries. section 10.1.1
- Add BOS and EOS markers, count token frequencies, retain the 50,000 most common entries, and reserve indexes for PAD and UNK. section 10.1.1
- Convert tokens to indexes for modeling, then reverse the mapping and remove boundary markers to reconstruct readable text. section 10.1.1
English–French tokenization
“I don't speak French.” becomes ['i</w>', 'don</w>', "'t</w>", 'speak</w>', 'fr', 'ench</w>', '.</w>']; “Je ne parle pas français.” becomes ['je</w>', 'ne</w>', 'parle</w>', 'pas</w>', 'franc', 'ais</w>', '.</w>']. section 10.1.1
import pandas as pd
df=pd.read_csv("files/en2fr.csv") #1
num_examples=len(df) #2
print(f"there are {num_examples} examples in the training data")
print(df.iloc[30856]["en"]) #3
print(df.iloc[30856]["fr"]) #4
Load the translation dataset and inspect paired English and French phrases before tokenization. section 10.1.1
