liveBook · Manning

Morpheme

As explained in LLMs in Production

A morpheme is the smallest unit of language that carries meaning or serves a grammatical function.

A morpheme is the smallest unit of language that carries meaning or serves a grammatical function. Prefixes and suffixes are familiar examples. A word can therefore contain meaningful parts smaller than the whole word. section 2.1.1

This matters when an LLM splits text into subword tokens for embeddings. Each token receives a dictionary index, a vector representation, and information about its position. But a useful vector is not a complete account of meaning: embeddings can place related words close together while still missing deeper meaning encoded in language. section 2.1.1

How LLMs in Production covers it

How deep it goes
Names prefixes and suffixes and sketches an embedding pipeline, without developing a method for analyzing word structure.
Examples
Uses prefixes and suffixes as examples of subword units.
What you can do after
Evaluate/understand why meaningful word parts matter to tokenization without assuming embeddings capture their full meaning.
How it connects
Connects morphemes to subword tokenization, semantic embeddings, and positional encoding.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.