chapter two

2 Large Language Models: A deep dive into language modeling

 

This chapter covers

  • Linguistic background for understanding meaning and interpretation
  • A comparative study on language modeling techniques
  • Attention and the transformer architecture
  • How Large Language Models both fit into and build upon these histories

The idiom “Once upon a time,” is how we signal to each other that a story is beginning. There isn’t an idiom for productionizing LLMs, so instead this chapter delves into linguistics as it relates to the development of Large Language Models (LLMs), exploring the foundations of semiotics, linguistic features, and the progression of language modeling techniques that have shaped the field of natural language processing (NLP). We will begin by studying the basics of linguistics and its relevance to LLMs in section 2.1, highlighting key concepts such as syntax, semantics, and pragmatics, that form the basis of natural language and play a crucial role in the functioning of LLMs. We will delve into semiotics, the study of signs and symbols, and explore how its principles have informed the design and interpretation of LLMs.

2.1 Language Modeling

2.1.1 Linguistic Features

2.1.2 Semiotics

2.1.3 Multilingual NLP

2.2 Language Modeling Techniques

2.2.1 N-Gram and Corpus-based techniques

2.2.2 Bayesian Techniques

2.2.3 Markov Chains

2.2.4 Continuous Language Modeling

2.2.5 Embeddings

2.2.6 Multilayer Perceptrons

2.2.7 RNNs and LSTMs

2.2.8 Attention

2.3 Attention is All You Need

2.3.1 Encoders

2.3.2 Decoders

2.3.3 Transformers

2.4 Really Big Transformers

2.5 Summary