Subword tokenization splits text into tokens that may be complete words, punctuation marks, or parts of words. It keeps frequent words whole while dividing less common or complex words into smaller components. Compared with word tokenization, this limits vocabulary growth; compared with character tokenization, tokens retain more meaning. section 8.2.1
A word can become two tokens
“It is unbelievably good!” can become ['It', 'is', 'un', 'believ', 'ably', 'good', '!']. In an English-to-French example, a pretrained tokenizer splits “French” into 'fr' and 'ench', and “français” into 'franc' and 'ais'. The resulting sequence mixes whole words with word fragments. section 8.2.1 section 10.1.1
Listing 10.1 A pretrained tokenizer
from transformers import XLMTokenizer #1
tokenizer = XLMTokenizer.from_pretrained("xlm-clm-enfr-1024")
tokenized_en=tokenizer.tokenize("I don't speak French.") #2
print(tokenized_en)
tokenized_fr=tokenizer.tokenize("Je ne parle pas français.") #3
print(tokenized_fr)
print(tokenizer.tokenize("How are you?"))
print(tokenizer.tokenize("Comment êtes-vous?"))
The XLMTokenizer example tokenizes English and French phrases with the same pretrained tokenizer. Inspect the printed tokens to see which words stay whole and which split. section 10.1.1
The XLM tokenizer uses '</w>' to mark a word boundary. It places no boundary between 'fr' and 'ench' because those tokens belong to the same word. To prepare translation data, tokenize each language, add BOS and EOS, and build a separate dictionary mapping its tokens to integer indexes. PAD and UNK handle padding and unknown tokens. Reverse dictionaries map indexes back to tokens for reconstruction. section 10.1.1
Indexed tokens can pass through word embeddings and positional encodings before an encoder-decoder Transformer processes them. A decoder then predicts translation tokens one at a time until EOS. Byte pair encoding (BPE) is another subword method: it repeatedly merges frequent adjacent character pairs to balance vocabulary size against token-sequence length. The detailed BPE procedure is not developed here; an existing implementation performs the encoding. section 10.0 section 11.2.1
