How deeply does How Large Language Models Work cover document segmentation?

Explains whitespace splitting, punctuation problems, and BPE’s iterative subword heuristic, but stops short of implementation details beyond a simple split.

Document segmentation

As explained in How Large Language Models Work

Text segmentation is the process of dividing text into a sequence of discrete substrings such as words or subwords.

Document segmentation divides a string into a sequence of smaller text pieces called tokens. In a language-model pipeline, it follows normalization and precedes mapping each token to a numeric identifier. A token may be a whole word, part of a word, or another discrete substring. section 2.2.1 section 2.6

The simplest approach splits wherever a space appears: “hello world” becomes “hello” and “world.” Punctuation exposes the weakness of this rule: “hello, world” can produce “hello,” and “world,” treating punctuation-attached text as a separate token from “hello.” section 2.2.3

From characters to subwords

Byte-pair encoding, or BPE, starts with individual characters, finds frequent adjacent pairs, replaces them with combined tokens, and repeats. A simplified run might find “ng,” replace its occurrences with a placeholder, add “ng” to the vocabulary, and continue until no useful common pairs remain. This reduces dependence on hand-written splitting rules and can represent words as reusable subword pieces. section 2.2.3

Segmentation is not a neutral cleanup step. Its choices help determine vocabulary size, while vocabulary size affects how well text can be represented and how much computation, memory, and storage are needed. Languages without spaces also challenge whitespace-based rules, and poorly chosen tokens can hinder arithmetic or word-game behavior. section 2.2.3 section 2.6

How How Large Language Models Work covers it

liveBook manning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Related concepts

tokenproduces · A token is a discrete unit of text, such as a word, word fragment, character, or symbol, represented and processed as an element by a language model.
tokenizationpart of · Tokenization is the process of dividing text or other sequential input into discrete units called tokens for computational processing.
image segmentationcontrasts with · Image segmentation is a computer-vision task that partitions an image and assigns labels to its pixels or regions to delineate objects or meaningful areas.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.