liveBook · Manning

Language model

As explained in Understanding AI

A large language model is a machine-learning model trained on extensive text data to generate and transform natural language.

A large language model, or LLM, is a machine-learning system that generates and transforms language by predicting what is likely to come next. Give it a question, an unfinished sentence, or an instruction, and it extends the text one token at a time. That simple loop can produce emails, explanations, summaries, conversations, and code. section 1.0 section 2.1

This can feel like understanding, but the useful distinction is prediction versus human-like thought. An LLM learns statistical patterns from enormous amounts of text: common word sequences, sentence structures, tones, and relationships expressed in context. It aims for a likely continuation, not guaranteed truth or human understanding, so a fluent answer can still be wrong. section 1.1.2 section 1.5 section 2.1

From a prompt to a response

For the prompt “The capital of France is…,” a model may assign a very high probability to “Paris,” a lower probability to “Lyon,” and almost none to “Banana.” After selecting a token, it adds that token to the context and repeats. A full answer emerges from many small predictions rather than from a complete plan prepared in advance. section 2.1

predict next word → add it → repeat

Text generation is an iterative loop: predict the next token, add it to the sequence, then use the expanded sequence to predict again. section 2.1

Several parts make this kind of prediction useful at scale:

  • Tokens break text into small pieces that the model can process; embeddings turn those pieces into numerical representations shaped by context. section 2.3
  • Attention lets each token weigh other relevant tokens in the visible sequence, helping the model maintain coherence and resolve relationships across passages. section 1.5 section 2.3
  • A context window limits what the model can currently see. Earlier conversation content outside that boundary is unavailable to its attention mechanism. section 2.0 section 2.3
  • Pretraining teaches broad language patterns, while fine-tuning and alignment shape a raw prediction engine into an assistant that follows instructions and reflects human preferences. section 2.2 section 2.3

The practical takeaway is to treat an LLM as a powerful pattern-based tool, not an all-knowing mind. Its outputs can be useful and natural-sounding, yet wording, prompt length, missing context, and imperfect guardrails can affect the result. Understanding the prediction process makes it easier to steer the tool and judge when an answer needs checking. section 2.0 section 2.3

How Understanding AI covers it

How deep it goes
Explains next-token generation, tokens, embeddings, attention, context windows, and assistant training without deep mathematics.
Examples
Uses the France capital completion, familiar phrase continuations, and raw-model continuations of a performance sentence.
What it assumes
Assumes no prior knowledge of tokens, embeddings, or context windows and introduces them from basic language prediction.
What you can do after
Diagnose and steer LLM responses by considering likelihood, prompt phrasing, context limits, and imperfect guardrails.
How it connects
Connects LLMs to tokens, embeddings, attention, transformers, context windows, pretraining, fine-tuning, alignment, and guardrails.

Also explained in

Compare how 89 books explain language model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Understanding AI. The section links point to the parts of the book it draws on.