2 Large language models: A deep dive into language modeling
This chapter covers
- The linguistic background for understanding meaning and interpretation
- A comparative study of language modeling techniques
- Attention and the transformer architecture
- How large language models both fit into and build upon these histories
If you know the enemy and know yourself, you need not fear the result of a hundred battles.
This chapter delves into linguistics as it relates to the development of LLMs, exploring the foundations of semiotics, linguistic features, and the progression of language modeling techniques that have shaped the field of natural language processing (NLP). We will begin by studying the basics of linguistics and its relevance to LLMs, highlighting key concepts such as syntax, semantics, and pragmatics that form the basis of natural language and play a crucial role in the functioning of LLMs. We will delve into semiotics, the study of signs and symbols, and explore how its principles have informed the design and interpretation of LLMs.