liveBook · Manning

Morphology

As explained in LLMs in Production

Morphology is the study of word structure and the ways words are formed from smaller meaningful units.

Morphology is the study of word structure and how words are formed from smaller meaningful units. It is one part of language, alongside phonetics, syntax, semantics, and pragmatics. section 2.1.1

In language modeling, morphology is associated with the tokenization layer. Tokenization can be a hurdle when too many <UNK> tokens appear, especially for specialized inputs such as code or math. section 2.5 section 2.5

How LLMs in Production covers it

How deep it goes
Identifies tokenization as the layer associated with morphology without detailing morphological segmentation.
Examples
Uses code and math as specialized settings where excessive <UNK> tokens can make tokenization difficult.
What you can do after
Evaluate/understand why tokenization can be a hurdle for specialized inputs.
How it connects
Connects morphology with tokenization, phonetics, syntax, semantics, and pragmatics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.