liveBook · Manning

Merge rule

As explained in Deep Learning with R, Third Edition

A token merge rule specifies an ordered operation for combining adjacent token symbols into a larger symbol.

A merge rule specifies an ordered operation for combining adjacent token symbols into a larger symbol. Repeatedly applying the most frequent pair builds subwords, from characters toward common words. section 14.2.2

Merging frequent pairs

Starting with character tokens, the frequent pair “o w” becomes “ow”; later, “ow” can combine with “l” to form “low”. The procedure records each pair in application order. section 14.2.2

How Deep Learning with R, Third Edition covers it

How deep it goes
Defines pair selection and merging, then implements vocabulary construction and ordered application with string substitution.
Examples
Uses toy word sequences, character-pair counts, progressive merges, and Moby Dick tokenization.
What it assumes
Starts with character splitting and reuses standardization and indexing while teaching pair counting and merging through executable steps.
What you can do after
Build a byte-pair encoding vocabulary and tokenizer from character tokens and an ordered merge list.
How it connects
Connects byte-pair encoding with subword tokenization, vocabularies, and tokenizers.
Cover of Deep Learning with R, Third Edition
liveBookmanning.com
Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.