liveBook · Manning

Language model

As explained in Build a Large Language Model (From Scratch)

A large neural language model trained on text to understand and generate language, typically using transformer self-attention.

A large language model (LLM) is a deep neural network trained on vast text collections to predict the next word or token, then generate and respond to language. Its apparent understanding means producing coherent, contextually relevant text rather than possessing human-like consciousness. section 1.0 section 1.1

The model’s size refers to both its adjustable parameters and its training dataset. Transformer self-attention lets it weigh tokens against one another, capturing contextual relationships and long-range dependencies. section 1.1 section 1.4

A practical LLM pipeline decomposes into these stages:

  • Prepare text by splitting it into word or subword tokens, applying byte pair encoding when appropriate, and forming input-output examples with a sliding window. section 2.0
  • Convert tokens into vectors that feed the model, whose decoder-only transformer predicts the next token from preceding context. section 1.1 section 2.0
  • Pretrain the GPT-like network on text, evaluate its language-modeling behavior, and fine-tune the resulting foundation model for instructions or target tasks. section 1.7

The original transformer combines an encoder, which produces contextual numerical representations, with a decoder, which generates output one word at a time. BERT-like models emphasize the encoder, while GPT-like LLMs use the decoder for generative tasks. section 1.4

From prompt to generated text

Given the input “This is an example” and the partial translation “Das ist ein,” a transformer decoder generates “Beispiel” as the next word. GPT-like models extend this next-token process to coherent text generation and can perform tasks in zero-shot or few-shot settings. section 1.4

Figure 1.9 The three main stages of coding an LLM are implementing the LLM architecture and data preparation process (stage 1), pretraining an LLM to create a foundation model (stage 2), and fine-tuning the foundation model to become a personal assistant or text classifier (stage 3).
figure

Use the three-stage map to navigate architecture and data preparation, pretraining a foundation model, and fine-tuning it into an assistant or classifier. section 1.7

How Build a Large Language Model (From Scratch) covers it

How deep it goes
Explains parameters, next-token prediction, tokenization, transformer modules, and training stages, while deferring detailed self-attention implementation.
Examples
Uses translation completion, ChatGPT-style assistants, GPT-3 datasets, token sampling, and zero-shot or few-shot task prompts.
What it assumes
Assumes general neural-network familiarity while teaching LLM structure, tokenization, data sampling, transformer roles, and training workflow step by step.
What you can do after
Build it from scratch by preparing text, implementing attention and a GPT-like model, pretraining it educationally, and fine-tuning task behavior.
How it connects
Connects large language models to deep learning, machine learning, transformers, self-attention, tokenization, pretraining, and fine-tuning.
Cover of Build a Large Language Model (From Scratch)
liveBookmanning.com
Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

Compare how 67 books explain language model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.