liveBook · Manning

SentencePiece

SentencePiece is a language-independent software library for learning and applying subword tokenization directly to raw text without whitespace-based pretokenization.

How 3 Manning books explain it

LLMs in Production

SentencePiece is a library and tokenization framework that learns subword vocabularies directly from raw text without requiring pretokenization by whitespace.

How it is used
Prepares task-specific text for LLM training, fine-tuning, and downstream data applications.
How deep it goes
Explains subword trade-offs and training parameters, but defers the internal algorithms behind SentencePiece.
Examples
Uses custom Python tokenizer training and arithmetic comparisons to show how tokenization choices affect model behavior.
What you can do after
Build it from scratch by training a tokenizer on project data and tuning vocabulary, frequency, and special-token settings.

Read how LLMs in Production explains SentencePiece →

Natural Language Processing in Action, Second Edition

SentencePiece tokenization is a subword tokenization technique that divides text into character-based pieces without requiring whitespace-delimited words.

How it is used
Uses subword units to encode natural language more efficiently for language-understanding and language-generation pipelines.
How deep it goes
Explains the motivation and benefits of subword encoding but stops short of algorithmic training or implementation details.
Examples
Uses prefixes, suffixes, syllables, compound words, named entities, characters, and graphemes as illustrative units.
What you can do after
Evaluate when subword or character-based units may represent language more efficiently than whole-word tokens.

Read how Natural Language Processing in Action, Second Edition explains SentencePiece →

Deep Learning with R, Third Edition

SentencePiece is a library for tokenizing text into subword units without requiring whitespace-delimited preprocessing.

How it is used
Preprocesses large text corpora into integer subword sequences for language-model training.
How deep it goes
Explains library use, vocabulary loading, round-trip conversion, and training-pipeline integration without detailing the tokenization algorithm’s internals.
Examples
Demonstrates tokenizing and detokenizing a fox sentence, loading a 32,000-term vocabulary, and preparing fixed-length training sequences.
What you can do after
Apply it via KerasHub to tokenize, detokenize, and prepare text for Transformer pretraining.

Read how Deep Learning with R, Third Edition explains SentencePiece →

Also covered in

What to know first

Related concepts

Questions about SentencePiece

What is SentencePiece?

SentencePiece is a language-independent software library for learning and applying subword tokenization directly to raw text without whitespace-based pretokenization.

Which Manning books explain SentencePiece?

LLMs in Production, Natural Language Processing in Action, Second Edition and Deep Learning with R, Third Edition.

What should I know before learning SentencePiece?

Manning books that teach SentencePiece usually introduce byte pair encoding, language model and T5 first.

What is the difference between SentencePiece and byte pair encoding?

SentencePiece is a language-independent software library for learning and applying subword tokenization directly to raw text without whitespace-based pretokenization. Byte-pair encoding is a subword tokenization method that builds a vocabulary by iteratively merging frequent adjacent symbols or tokens.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.