How deeply does LLMs in Production cover SentencePiece?

Explains subword trade-offs and training parameters, but defers the internal algorithms behind SentencePiece.

SentencePiece

SentencePiece is a library and tokenization framework that learns subword vocabularies directly from raw text without requiring pretokenization by whitespace.

SentencePiece is a subword tokenization framework for converting text into model-ready numbers. It uses a smaller vocabulary than word tokenization while preserving more meaning than character tokenization, and it explicitly handles whitespace. section 4.4 section 4.4.1

The practical advantage is control: training a tokenizer on task-specific data can reduce unknown tokens and improve encoded semantics. The right vocabulary still depends on the task and the model being trained or fine-tuned. section 4.4 section 4.4.1

When training a SentencePiece tokenizer, tune the parameters that determine coverage, dictionary quality, and model-training compatibility:

Increase vocab_size for broader language coverage and robustness, while accepting greater computational complexity. section 4.4.1
Set min_frequency to exclude very rare subwords, balancing dictionary size against potentially useful rare tokens. section 4.4.1
Define special_tokens such as start, padding, end, unknown, and mask markers required by the training setup. section 4.4.1

Training on project data

A Python workflow collects text files from a data directory, instantiates a tokenizer, and trains it with a chosen vocabulary size, minimum frequency, progress display, and special-token list. The same workflow pattern supports preparing a tokenizer for a specific dataset rather than adopting an unrelated default. section 4.4.1

Table 4.3 Tokenization allows GOAT 7B to outperform GPT-4 in math

Prompt	GOAT 7B	GPT-4 1.7T	Correct
3978640188 + 42886272 =	4021526460	4,021,526,460	4,021,526,460
4523646 minus 67453156	–62929510	–63,930,510	–62,929,510
Calculate 397 × 4429	1758313	1,757,413	1,758,313
What is 8914/64?	139 R 18	139.15625	139.28125 Or 139 R 18

Compare arithmetic outputs where tokenization quality becomes a model-performance factor, including digit-sensitive representations and competing answers. section 4.4.1

How LLMs in Production covers it

liveBook manning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Natural Language Processing in Action, Second EditionSentencePiece tokenization is a subword tokenization technique that divides text into character-based pieces without requiring whitespace-delimited words.
liveBook manning.com

Deep Learning with R, Third EditionSentencePiece is a library for tokenizing text into subword units without requiring whitespace-delimited preprocessing.

Compare how 3 books explain SentencePiece →

Related concepts

tokenizationimplements · Tokenization is the process of dividing text or other sequential input into discrete units called tokens for computational processing.
subword tokenizationuses · Subword tokenization is a method of representing text with tokens that may be whole words or smaller word fragments, allowing unfamiliar words to be encoded as combinations of units.
byte pair encodingalternative to · Byte-pair encoding is a subword tokenization method that iteratively merges frequent adjacent symbol pairs to build a vocabulary of larger units.
KerasHubused by · KerasHub is a Keras library providing reusable components, pretrained models, and utilities for machine-learning and generative-modeling tasks.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.