SentencePiece
SentencePiece is a language-independent software library that learns and applies subword tokenization directly to raw text without requiring whitespace-based pretokenization.
LLMs in Production
SentencePiece is a library and tokenization framework that learns subword vocabularies directly from raw text without requiring pretokenization by whitespace.
- How it is used
- Prepares task-specific text for LLM training, fine-tuning, and downstream data applications.
- How deep it goes
- Explains subword trade-offs and training parameters, but defers the internal algorithms behind SentencePiece.
- Examples
- Uses custom Python tokenizer training and arithmetic comparisons to show how tokenization choices affect model behavior.
- What you can do after
- Build it from scratch by training a tokenizer on project data and tuning vocabulary, frequency, and special-token settings.
Natural Language Processing in Action, Second Edition
SentencePiece tokenization is a subword tokenization technique that divides text into character-based pieces without requiring whitespace-delimited words.
- How it is used
- Uses subword units to encode natural language more efficiently for language-understanding and language-generation pipelines.
- How deep it goes
- Explains the motivation and benefits of subword encoding but stops short of algorithmic training or implementation details.
- Examples
- Uses prefixes, suffixes, syllables, compound words, named entities, characters, and graphemes as illustrative units.
- What you can do after
- Evaluate when subword or character-based units may represent language more efficiently than whole-word tokens.
Read how Natural Language Processing in Action, Second Edition explains SentencePiece →
Deep Learning with R, Third Edition
SentencePiece is a library for tokenizing text into subword units without requiring whitespace-delimited preprocessing.
- How it is used
- Preprocesses large text corpora into integer subword sequences for language-model training.
- How deep it goes
- Explains library use, vocabulary loading, round-trip conversion, and training-pipeline integration without detailing the tokenization algorithm’s internals.
- Examples
- Demonstrates tokenizing and detokenizing a fox sentence, loading a 32,000-term vocabulary, and preparing fixed-length training sequences.
- What you can do after
- Apply it via KerasHub to tokenize, detokenize, and prepare text for Transformer pretraining.
Read how Deep Learning with R, Third Edition explains SentencePiece →
Related concepts
- tokenizationimplements · Tokenization is the process of dividing text or other sequential input into discrete units called tokens for computational processing.
- subword tokenizationuses · Subword tokenization is a method of representing text with tokens that may be whole words or smaller word fragments, allowing unfamiliar words to be encoded as combinations of units.
- byte pair encodingalternative to · Byte-pair encoding is a subword tokenization method that iteratively merges frequent adjacent symbol pairs to build a vocabulary of larger units.
- KerasHubused by · KerasHub is a Keras library providing reusable components, pretrained models, and utilities for machine-learning and generative-modeling tasks.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.