Byte pair encoding (BPE) is a subword tokenization algorithm. It iteratively merges the most frequent character or character-sequence pairs until it forms a fixed-size vocabulary, then maps resulting units to token IDs for decoder-only transformers. section 3.2
BPE addresses practical tokenization problems by:
- Reducing vocabulary pressure and the number of possible word forms, especially in morphologically rich languages. section 3.2
- Representing rare or complex words as reusable subword units rather than requiring every complete word in the vocabulary. section 3.2
- Combining Unicode-safe byte encoding, regular-expression token splitting, BPE merges, and vocabulary lookup in an input encoder. section 3.2
Subword decomposition
The word “inventing” might be split into “in,” “vent,” and “ing.” These units can be recombined across related or previously unseen words. section 3.2
Listing 3.1 Input encoder used in decoder-only transformers (BPE abstracted)
class Encoder:
def __init__(self, encoder, bpe_merges, errors='replace'):
self.encoder = encoder
self.decoder = {v: k for k, v in encoder.items()}
self.byte_encoder = bytes_to_unicode()
self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges))))
self.cache = {}
self.errors = errors
self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+|
?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""")
def bpe(self, token):
... #1
def encode(self, text):
bpe_tokens = [] #2
for token in re.findall(self.pat, text): #3
byte_seq = ''.join(self.byte_encoder[b] for b in
token.encode('utf-8')) #4
token_encode = byte_seq
bpe_tokens.extend(
self.encoder[bpe_token] for bpe_token in
self.bpe(byte_seq).split(' ') #5
)
return bpe_tokens
def decode(self, tokens):
text = ''.join([self.decoder[token] for token in tokens]) #6
return text
def __convert_back_to_original_UTF_8_string_using_byte_decoder
(self, bytearray):
text = bytearray([self.byte_decoder[c]
for c in text]).decode('utf-8', errors=self.errors) #7
return text
def get_encoder(model_name, models_dir):
with open(os.path.join(models_dir, model_name,
'encoder.json'), 'r') as f: #8
encoder = json.load(f)
with open(os.path.join(
models_dir, model_name,
'vocab.bpe'), 'r',
encoding='utf-8') as f:
bpe_data = f.read()
bpe_merges = [tuple(merge.split())
for merge in bpe_data.split('\n')[1:-1]] #9
return Encoder(encoder=encoder, bpe_merges=bpe_merges)
Inspect the abstracted encoder showing byte conversion, token matching, merge ranks, caching, and vocabulary-ID conversion around the BPE step. section 3.2
