SentencePiece treats text as reusable subword pieces rather than assuming that human words are indivisible units. This matters when spaces do not cleanly mark meaningful boundaries, compound words contain internal structure, or unfamiliar words need to be represented through related pieces. section 2.2
SentencePiece is useful because it can:
- Split words into meaningful parts such as prefixes, suffixes, and interior syllables, allowing patterns learned from one word to transfer to another. section 2.2
- Represent text with characters, graphemes, or other smaller pieces when word-level units are ambiguous or vocabularies are large. section 2.2
- Support natural-language understanding of new words and natural-language generation of newly formed expressions. section 2.2
From words to pieces
Instead of treating a word as one opaque token, imagine analyzing it as pieces such as “pre,” “la,” and “fix.” The same strategy can preserve useful structure in compounds and named entities while avoiding a vocabulary made entirely of whole words. section 2.2
