liveBook · Manning

Morphology

As explained in LLMs in Production

Morphology is the study of word structure and the ways words are formed from smaller meaningful units.

A word has structure: smaller meaningful units can combine to form it. Morphology is the study of that structure and how words are formed. section 2.1.1

For a large language model, morphology enters through the tokenization layer. It is one of five linguistic components, alongside phonetics, syntax, semantics, and pragmatics. section 2.5

Tokenization is worth checking when a model encounters specialized text. Too many <UNK> tokens can be a major hurdle for code or math. That is a tokenization warning, not a worked example of splitting words into meaningful units. section 2.5

How LLMs in Production covers it

How deep it goes
Identifies morphology as a linguistic component and assigns it to tokenization, without working through word structure or a tokenizer implementation.
Examples
Uses code and math as cases where excessive <UNK> tokens hinder tokenization, without giving a worked morphology example.
What you can do after
Evaluate/understand morphology’s place in LLM tokenization and recognize excessive <UNK> tokens as a potential obstacle.
How it connects
Connects morphology to tokenization and to phonetics, syntax, semantics, and pragmatics.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.