Chapter 2: Large language models: A deep dive into language modeling

Introduction

This introduction maps the chapter’s path from linguistic foundations and earlier language models to transformers and LLMs, and explains why that background matters in production.

• Linguistic foundations roadmap: Describe the chapter’s plan to examine syntax, semantics, pragmatics, and semiotics as background for understanding how LLMs handle language and meaning.
  Why it matters: These topics provide context for the language-modeling techniques that follow.
• Language model evolution roadmap: Trace the approaches the chapter plans to compare, from N-grams and naive Bayes through neural network methods to transformers and LLMs. State the introduction’s characterization of LLMs as large transformer-based models that build on and surpass earlier techniques.
  Why it matters: The historical comparison frames what is distinctive about today’s LLMs.
• Production value of fundamentals: Justify the book’s claim that understanding LLM technology and its history helps engineers build products more effectively than simply deploying a large model with more resources.
  Why it matters: Technical context can help you anticipate production gotchas instead of relying on brute force.

Key terms: syntax, semantics, pragmatics, semiotics, N-grams, naive Bayes classifiers, multilayer perceptrons (MLPs), recurrent neural networks (RNNs), long short-term memory (LSTM) networks, transformer-based models

2.1 Language modeling

The section uses layers of abstraction to explain why conveying an idea to an LLM is difficult and introduces the language features the chapter will examine.

• Language and computation abstractions: Explain how the section compares the path from feelings and thoughts to language, math, and binary with the layers between human-readable programming languages and the binary a computer reads. Describe abstraction as a move away from a source that can lose information, not a claim that every layer literally descends from the previous one.
  Why it matters: This comparison gives you a way to reason about why an intended meaning can be hard to convey to a model.
• Meaning loss at intermediary steps: Explain why each abstraction layer can be a point where an intended meaning is changed or lost, and why tokenization and embeddings matter even when the code and the original expression are reliable.
  Why it matters: An LLM can receive something different from what its user intended before the model itself processes it.
• Language features as a starting point: Describe why the chapter starts with features of language before going deeper into language-modeling techniques: their interactions shape communication and help build an intuition for the challenges models face. Name examples the section previews, such as phonetics, pragmatics, morphology, syntax, and semantics.
  Why it matters: Considering language itself helps explain challenges that are easy to miss when focusing only on model code.

Key terms: abstraction, language, math, binary, assembly, disconnect point, failure points, tokenization, embeddings, linguistic features

2.1.1 Linguistic features

The section explains five linguistic features and how each affects what language models can represent, learn, and miss.

• Phonetic information and representation: Explain what phonetics contributes to communication, why text omits some of it, and why representing speech sounds in an LLM remains difficult despite tools such as the International Phonetic Alphabet.
  Why it matters: The sound and tone of speech can change a message’s meaning even when its words stay the same.
• Syntax and contextual ambiguity: Distinguish grammatical structure and word order from meaning, and explain why interpreting a grammatically valid sentence can require semantic and pragmatic context.
  Why it matters: Producing well-formed sentences is not the same as resolving what those sentences mean.
• Changing meanings and embeddings: Describe semantics as encoded word meaning, explain that meanings shift through narrowing, broadening, or reinterpretation, and distinguish semantic embeddings from a complete representation of meaning.
  Why it matters: A model’s representation of words must contend with meanings that change and cannot be fully captured by vector proximity.
• Pragmatic context beyond text: Explain how shared culture, lived experience, and other out-of-text context shape an utterance’s interpretation, and distinguish a model’s learned responses to such cues from lived knowledge of society.
  Why it matters: Much of what people take an utterance to mean is not stated in its words.
• Dataset bias and entailment: Explain how training data inevitably carries pragmatic bias, how explicit examples and entailed information differ, and how curation, instruction data, or prompting can steer the responses a model produces.
  Why it matters: A model can respond reliably to a task only to the extent that its data and prompt supply useful examples and cues.
• Morphemes and tokenization choices: Distinguish meaningful word parts from model tokens, and explain how a tokenizer that splits text poorly or produces too many unknown tokens can discard information needed for a task.
  Why it matters: Tokenization determines which parts of an input the model can use at all.

Key terms: phonetics, phonology, International Phonetic Alphabet, IPA, tone, syntax, word order, syntactic ambiguity, semantics, pragmatics

2.1.2 Semiotics

The section uses semiotics to show how people assemble meaning from multiple kinds of cues, why LLMs have an incomplete version of that process, and how prompting can help.

• Semiotic model of meaning: Explain how the adapted Peircean semiotic triangles organize meaning-making into a sequence of firstness, secondness, and thirdness that leads toward interpretation, while also modeling individual pieces of the process.
  Why it matters: This model provides a way to notice which parts of meaning-making an LLM can and cannot draw on.
• Cues needed for interpretation: Describe how meaning depends on more than words alone, including visual and written structure, literal meanings, organizers such as tables, and cultural habits or scripts. Explain the section's claim that LLMs do not have direct access to the full range of human cues and experience, which limits their interpretations.
  Why it matters: Considering missing cues helps explain why generating plausible language is not the same as interpreting it as a person does.
• Prompted script reproduction: Explain why explicitly naming a familiar script in a prompt can improve an LLM's output: a foundation model trained on many examples can reproduce that kind of script, as with a request for a step-by-step explanation.
  Why it matters: It connects the section's account of social scripts to a practical way of guiding model output.
• Recursive clarification of meaning: Describe how people can resolve unclear meaning by asking clarifying questions and repeating the interpretive process, and contrast that with the section's claim that current models generally need purposeful prompting to do this.
  Why it matters: Clarification offers a concrete example of meaning-making as an iterative process rather than a one-pass response.

Key terms: semiotics, Peircean semiotic triangle, firstness, secondness, thirdness, interpretation, orthography, semantics, ritual, scripts

2.1.3 Multilingual NLP

The section argues that multilingual access and contested language boundaries matter in LLM design, then contrasts localization with internationalization as early design directions.

• Multilingual reach and language boundaries: Explain why an English-only LLM excludes many potential users and customers, and why boundaries between languages and dialects can be unreliable and harmful when designing a system.
  Why it matters: A model's language choices affect both who can use it and whose voices it may leave out.
• Early localization or internationalization choice: Distinguish localization, illustrated by Mozilla's language-specific browser versions, from internationalization, illustrated by Ikea's word-light, picture-based instructions. Explain why choosing a direction at the start can make later expansion less costly and help avoid excluding users.
  Why it matters: Making this choice early can turn translation and formatting from a late expense into a planned investment.
• Linguistics to computational modeling: Describe the section's transition from linguistic considerations to evaluating computational language-modeling techniques: language structure is distinct from meaning, and the upcoming techniques will be examined for how they represent language and address linguistic features or meaning.
  Why it matters: This distinction gives the reader a question to carry into the chapter's discussion of modeling techniques.

Key terms: multilingual models, Anglocentric, languages and dialects, localization, l10n, internationalization, i18n, structure of language, meaning, linguistic features

2.2 Language modeling techniques

This section introduces the processing choices common to language models, then uses bag-of-words to show both the value and the limits of a simple baseline.

• Shared language modeling choices: Describe the three choices every language model needs: how to divide text into tokens, how to convert tokens to numbers and back, and how the model will process the resulting input.
  Why it matters: These choices provide a way to compare the techniques introduced in the rest of the section.
• Bag-of-words frequency baseline: Explain how a bag-of-words (BoW) model counts word occurrences in a vocabulary and how those counts can offer clues about a text, such as themes in a president’s inaugural speech.
  Why it matters: Even a simple count-based model can reveal patterns and provide a baseline for evaluating other models.
• Bag-of-words limitations and successor: Explain that BoW loses word order and context, does not capture features such as semantics or phrases, and is sensitive to tokenization; state that n-gram models were conceived to address its lack of sequence data.
  Why it matters: The limitations of BoW explain why language modeling techniques move beyond word counts.

Key terms: token, tokenized inputs, numerical values, bag-of-words (BoW), vocabulary, CountVectorizer, frequency, baseline, whitespace, tokenization

2.2.1 N-gram and corpus-based techniques

This section introduces N-gram language models as practical, frequency-based baselines, shows how a trigram generates text, and explains why fixed-length context cannot capture all aspects of language.

• N-gram context and probabilities: Explain how N-grams improve on bag-of-words by using an N-length word context to make probabilistic connections between words, rather than modeling linguistic features such as syntax.
  Why it matters: This distinction explains both why N-grams are useful and why their predictions have limits.
• Trigram generation workflow: Describe the example's workflow: clean and minimally pad or format text, train a trigram model with everygrams, and supply up to two tokens as context when generating further text.
  Why it matters: The workflow connects the statistical idea to a small working text generator.
• Fixed-length modeling limitations: Explain why a fixed N misses phrases of different lengths and why simply making N enormous is not a practical solution; distinguish this limitation from the model's failure to encode individual-word semantics.
  Why it matters: These limits explain why a useful baseline is not a complete model of language.
• Baseline and evaluation value: Describe N-grams as useful baselines for text analysis, particularly when an analyst already knows an utterance's pragmatic context, while noting that frequency-based probability, log-score, entropy, or perplexity calculations can poorly reflect real-world language.
  Why it matters: It clarifies how to use N-gram results without overinterpreting their scores.

Key terms: N-gram, BoW, context, probabilistic connections, linguistic features, trigram, everygrams, padding, tokens, Listing 2.1

2.2.2 Bayesian techniques

This section introduces Bayes’ theorem as a basis for text classification, then explains why the simple Bayesian model shown is limited for language generation and sequence modeling.

• Bayesian probability relationship: Explain how Bayes’ theorem relates the probability of a hypothesis given evidence to the probability of that evidence given the hypothesis, the prior probability of the hypothesis, and the probability of the evidence.
  Why it matters: This relationship explains what the classifier is estimating when it evaluates a hypothesis about text.
• Bayesian text classification role: Describe the section’s simple naive Bayes language model as a text classifier that uses prior-learned probabilities to evaluate a hypothesis, rather than as a model that effectively generates language. State that it can augment a generative model.
  Why it matters: The distinction prevents a useful classification technique from being mistaken for a text generator.
• Static words and disconnected sequences: Explain why assigning words static Bayesian values is a poor fit for context-dependent words such as “it,” and why treating sequences as unconnected loses word-order information compared with N-grams.
  Why it matters: These limitations motivate the section’s move back toward sequence modeling.

Key terms: Bayes’ theorem, hypothesis, evidence, prior, naive Bayes, classification, generative language models, Listing 2.2, LogPrior, LogLikelihood

2.2.3 Markov chains

The section explains how Markov models generate language from state-dependent probabilities, where they are useful, and why their limited context motivates continuous models.

• State dependent word probabilities: Explain how a Markov language model uses its current state to assign probabilities to what comes next, rather than relying only on words’ overall historical frequencies.
  Why it matters: State-dependent probabilities are the central idea behind the model described here.
• Corpus based Markov generation: Describe the section’s generative example as training on a document of utterances tokenized at whitespace, and identify the text-processing or prediction tasks for which Markov models are used.
  Why it matters: The example connects state-based probabilities to generated text and practical uses.
• Descriptive language modeling: Distinguish describing language as it appears in a corpus from prescribing how language ought to behave, and explain which approach the section associates with effective language modeling.
  Why it matters: This distinction explains why the section models observed language rather than imposing rules for correct language.
• Current state context limits: Explain why a Markov model’s current state cannot effectively represent broader historical or societal context or word meaning, and how that limitation motivates continuous models with semantic embeddings.
  Why it matters: This limitation motivates the next family of language-modeling techniques.

Key terms: Markov chains, hidden Markov models, state, stochasticity, N-gram models, generative language model, whitespace, tokenizer, document, part-of-speech tagging

2.2.4 Continuous language modeling

The section uses a continuous bag-of-words model to show how surrounding words can produce embeddings, while introducing a neural-network training loop and its activation choices.

• CBoW context window prediction: Explain how CBoW uses words on both sides of a target to predict the word in the middle, and why that setup is more useful for embeddings than for generating text.
  Why it matters: The words a model can see determine both the context its embeddings capture and the predictions it can make.
• Continuous technique and model choices: Distinguish the continuous modeling technique from the particular models used to demonstrate it: this example combines BoW processing with a simple neural network, but those components could be replaced.
  Why it matters: The example's simple components illustrate an approach rather than prescribe one fixed architecture.
• Neural network training loop: Trace the training-loop pieces highlighted in the CBoW implementation: weight initialization, ReLU activation, a final softmax layer, and forward and backpropagation within gradient descent.
  Why it matters: These steps recur even when frameworks hide their implementation.
• ReLU benefits and limitations: Explain why the example uses ReLU despite its limitations: it is intuitive and easy to code and helps with vanishing gradients, but does not solve exploding gradients and discards negative comparisons; the section names ELU and GEGLU/SWIGLU as situational alternatives.
  Why it matters: An activation that is convenient for a teaching example is not necessarily the best choice for every language model.

Key terms: continuous bag-of-words (CBoW), context window, target word, embeddings, continuous modeling, BoW model, neural network, N-grams, naive Bayes, model weights

2.2.5 Embeddings

This section explains how embeddings derive useful semantic relationships from word usage, where they fall short of human understanding, and how visualization can help assess them.

• Meaning from word usage: Explain how embeddings turn tokenized vectors without inherent meaning into representations shaped by observed text patterns, including words or subwords appearing in similar contexts. Describe the Word2Vec king–man+woman example as an illustration of a learned semantic relationship.
  Why it matters: This explains why continuous language models can work with relationships between words rather than just their token identities.
• Usage versus pragmatic context: Distinguish semantic patterns embeddings learn from text usage from the pragmatic context humans use to interpret an utterance. Explain why more representative data might improve embeddings, while better embeddings can themselves improve model performance.
  Why it matters: The distinction helps prevent useful word relationships from being mistaken for complete human-like understanding.
• Embedding visualization and validation: Describe how plotting embeddings with pyplot can support explainability and validation during pretraining: semantically similar embeddings appearing near one another is an encouraging sign. State that the section's CBoW visualization was successful but sparse, and identifies denser semantic representations as an area for improvement.
  Why it matters: A visual check can reveal whether training is producing plausible relationships between words.

Key terms: embeddings, tokenized vectors, collocation, Word2Vec, nearest neighbor, semantics, pragmatics, representative data, pyplot, model explainability

2.2.6 Multilayer perceptrons

This section presents MLPs as flexible neural-network building blocks while emphasizing that an LLM’s capabilities depend on how data moves through its architecture, not just on its layer count.

• Feature composition in MLPs: Explain how an MLP combines weights and biases that detect features to detect larger, more complex features, and why MLPs are building blocks in many neural-network architectures.
  Why it matters: This gives the reader a starting point for understanding what layers contribute to larger models.
• Configurable layers and architecture: Distinguish the PyTorch MLP class, whose number of hidden layers is set when instantiated, from the earlier two-layer CBoW implementation. Explain why adding layers or parameters alone does not necessarily improve a model: architectures also differ in how data enters and moves through their layers.
  Why it matters: It prevents equating model size with capability and prepares the reader to compare later architectures.

Key terms: MLP, weights, biases, features, fully connected layers, Listing 2.6, num_hidden_layers, CBoW, RNNs, CNNs

2.2.7 Recurrent neural networks and long short-term memory networks

This section explains how RNNs and LSTMs model sequences, compares their sentiment-analysis performance, and shows why their computational costs motivate attention.

• RNN sequence memory and gradients: Explain how an RNN processes an ordered sequence using an internal state, allowing variable-length inputs, and why long sequences can cause vanishing or exploding gradients.
  Why it matters: This explains both what RNNs improved over earlier models and why LSTMs were introduced.
• LSTM memory and bidirectionality: Explain that an LSTM is an RNN variant using memory cells and gates to better handle sequences of different lengths, and describe bidirectional processing as combining left-to-right and right-to-left outputs for context.
  Why it matters: These changes help explain why the LSTM handles the section's sequence tasks better than the basic RNN.
• Sentiment example preparation and configuration: Describe the sentiment-analysis example's word tokenization and lemmatization, padding of variable-length sequences to a maximum length, and the LSTM configuration options highlighted alongside the RNN: layers, bidirectionality, and dropout.
  Why it matters: These details connect the architectural discussion to the models and data used in the example.
• LSTM performance comparison: State what the example reports about LSTM versus RNN training and validation accuracy and epochs, and distinguish fewer epochs to reach performance from faster individual training.
  Why it matters: The reported results show the benefit of the LSTM without implying that it is cheaper to train.
• Recurrent model compute tradeoff: Explain that LSTMs remain resource-heavy, particularly on long, detailed problems; bidirectionality and dropout raise training costs, motivating attention as a way to focus on influential sequence parts and ignore less useful ones.
  Why it matters: This tradeoff provides the section's transition from recurrent models to attention.

Key terms: RNN, sequence, internal state, variable length inputs, vanishing gradient, exploding gradients, LSTM, memory cells, gating mechanisms, bidirectional

2.2.8 Attention

This section explains how attention uses queries, keys, and values to select relevant context, distinguishes attention patterns, and introduces its benefits and limits before transformers.

• Query key value scoring: Explain how attention compares a Query with Keys using dot-product similarity, then uses the resulting attention scores with Values to determine how much attention to pay to parts of a sequence.
  Why it matters: This is the basic mechanism behind the attention models introduced next.
• Attention patterns and masking: Distinguish bidirectional self-attention, which relates words across the same sentence, from causal attention, which considers only preceding words. Describe masking as hiding part of a sequence and making the model guess what is hidden.
  Why it matters: Different attention patterns determine which context a language model can use.
• Attention benefits and remaining limits: Explain why the section presents attention as an advance over older sequence techniques: it helps select relevant input and addresses slow LSTM training while retaining performance with few epochs. State that speed and memory problems remain, including quadratic scaling, motivating later approaches and transformers.
  Why it matters: Attention is a major advance, but its remaining costs help explain the move to newer architectures.

Key terms: Query, Key, Value, dot product similarity, attention score, dot product attention, bi-directional self-attention, causal attention, masked attention, context windows

2.3 Attention is all you need

Introduces transformers as a shift from earlier NLP techniques to an architecture based on attention rather than recurrence or convolution.

• Attention-only transformer approach: Explain that Vaswani and colleagues proposed using attention without recurrence or convolution, while specifying more carefully where the queries, keys, and values come from.
  Why it matters: This is the central architectural shift behind the transformer mechanisms discussed next.
• Transformer transition in NLP: Describe how the section places transformers after rule-based, statistical, and neural-network approaches, and presents them as an innovation associated with stronger language models.
  Why it matters: The transition explains why the chapter moves from earlier language-modeling techniques to transformers.

Key terms: attention, recurrence, convolutions, Q, K, and V, transformer architecture, rule-based methods, statistical models, neural networks, state-of-the-art language models

2.3.1 Encoders

This section explains how transformer encoders preserve useful token meanings while incorporating position, then describes BERT as an encoder-only model and weighs encoder strengths against their costs.

• Encoder representations and position: Explain that an encoder is the first half of a full transformer and uses token embeddings, self-attention, and positional encoding to represent both meaning and position without losing too much semantic similarity.
  Why it matters: This explains how an encoder can distinguish where a token appears while retaining a useful representation of what it means.
• BERT as an encoder example: Describe BERT as an encoder-only transformer whose embeddings could be adapted quickly to new tasks and data with minimal training, outperforming Word2Vec embeddings by the section’s account.
  Why it matters: BERT gives a concrete example of why encoder representations proved useful beyond a single task.
• Encoder strengths and costs: Distinguish encoders’ strengths in classification and other understanding-oriented tasks, long-range dependency modeling, and parallel processing from their need for substantial data and architectural complexity.
  Why it matters: These trade-offs help explain when an encoder is attractive and what it takes to use one effectively.

Key terms: encoder, embeddings, tokens, self-attention, positional encoding, semantic similarity, BERT, encoder-only transformer, Word2Vec, few-shot learning

2.3.2 Decoders

This section describes decoder structure, why decoders suit language generation, and where syntax-focused generation can fail.

• Decoder structure and role: Describe the decoder as the second half of a full transformer, following an encoder, and identify its base form as having two multihead attention blocks and three sum-and-normalize layers.
  Why it matters: This places decoders within the transformer architecture before considering decoder-only models.
• Decoder language generation strengths: Explain that decoders excel at natural language generation rather than the natural language understanding tasks associated with encoders. Describe next-token generation using already-generated tokens, and give GPT as a decoder-only example.
  Why it matters: Next-token generation explains both decoder-only language models and why their output can be streamed.
• Syntax and meaning limitations: Explain that the section associates decoder generation with learning and applying syntax, but warns that syntax-only generation can miss intended meaning and produce hallucinations.
  Why it matters: Fluent sentence generation does not guarantee that the output means what was intended or is reliable.

Key terms: decoder, encoder, multihead attention blocks, sum and normalize layers, natural language generation (NLG), natural language understanding (NLU), next token, shifted right, decoder-only, Generative Pre-trained Transformer (GPT)

2.3.3 Transformers

This section explains how a full transformer combines an encoder and decoder for conditional generation, then discusses its practical tradeoffs, linguistic limitations, and the value of finetuning pretrained models.

• Encoder decoder cooperation: Explain how a full transformer passes the encoder’s understanding of an input into the decoder’s second multihead attention block, allowing the decoder to generate output conditioned on that input.
  Why it matters: This explains why a full transformer suits tasks that require understanding something before generating a response.
• Transformer performance tradeoffs: Describe the section’s comparison of transformer speed and resource demands: parallelized inputs make transformers fast relative to LSTMs, but training requires substantial data and VRAM.
  Why it matters: Speed is an important advantage, but it does not make transformers cheap to train.
• Limits of linguistic modeling: Distinguish the linguistic abilities the section attributes to typical models from those it says they generally do not use: syntax-heavy modeling and semantic approximations rather than pragmatics, phonetics, or context-aware morphology.
  Why it matters: Strong outputs do not necessarily mean a model represents every aspect of language.
• Pretraining and task finetuning: Explain why finetuning a pretrained foundational model such as BERT on representative task or domain data can improve its performance without training from scratch: pretrained weights already reflect learning from a much larger dataset, so less new data is needed.
  Why it matters: This connects general pretrained models to practical improvements on a specific task or domain.

Key terms: full transformer, encoder, decoder, second multihead attention block, conditional generation, T5, parallelization of inputs, LSTMs, memory intensive, VRAM

2.4 Really big transformers

This section explains what larger transformers make possible, where their language abilities fall short, and the practical risks of using them.

• Scale and emergent abilities: Explain how increasing transformer model and training scale can enable performance and tasks that smaller language models cannot achieve, including emergent behavior.
  Why it matters: Model size helps explain both the appeal of LLMs and the costs of running them.
• Prompting and task adaptation: Distinguish finetuning from using examples in a prompt, and describe few-shot, one-shot, and zero-shot prompting. Explain why accomplishing specialized tasks with little or no labeled task data makes LLMs useful.
  Why it matters: This distinction helps you decide whether a task needs model adaptation or can be attempted through prompting.
• Context and linguistic limits: Explain how large and diverse training data and longer input contexts help LLMs handle ambiguity, varied writing, and long-range context. Distinguish those improvements from actually possessing semantic understanding or a built-in pragmatic system.
  Why it matters: Better responses do not necessarily mean the model understands meaning or a real-world situation.
• Application fit by language demands: Apply the section’s syntax, semantics, and pragmatics distinctions to explain why writing assistants and chatbots are comparatively easy, shopping assistants need supplied product context, and chess remains difficult.
  Why it matters: Matching a product idea to what the model can actually do helps avoid choosing a deceptively hard use case.
• Compute and runtime constraints: Describe the computational and energy costs of foundational LLMs and explain what the Bloom example demonstrates about running a large model versus a smaller variant.
  Why it matters: A model’s capabilities matter only if you can afford to train and run it on available hardware.
• Training data bias risks: Explain how biases in large real-world training datasets can appear in generated outputs, using the section’s “leader” image example, and why the section treats unrepresentative output as an accuracy problem.
  Why it matters: Checking whether outputs represent the intended world is part of judging model quality, not just fluency.

Key terms: LLMs, transformer-based models, parameters, emergent behavior, transfer learning, finetuning, few-shot prompting, one-shot prompting, zero-shot prompting, input space

2.5 Summary

This summary connects linguistic features and language-modeling advances to modern LLM architectures and the effects of model scale.