liveBook · Manning

Language model

As explained in Build a DeepSeek Model (From Scratch)

A neural language model that learns statistical patterns from large text corpora to generate and understand human language.

A language model learns statistical relationships from vast text datasets and uses fixed parameters to predict the next token, generating human-language sequences during inference. section 2.1.1

Modern language models combine learning phases with Transformer-based computation:

  • Pretraining adjusts model weights over vast datasets to learn grammar, facts, reasoning patterns, and relationships between words. section 2.1.1
  • Inference uses fixed weights to generate text without further learning. section 2.1.1
  • A standard Transformer layer combines multi-head self-attention with a feed-forward neural network. section 1.2.1
  • Scaled architectures replace these components with alternatives such as Multi-Head Latent Attention and Mixture-of-Experts to address attention efficiency and model capacity. section 1.2.1

Inference after pretraining

Asking a model to make a travel plan for Italy illustrates inference: the model does not update its parameters, but predicts successive tokens using what pretraining taught it. section 2.1.1

Figure 1.4 A simplified view of the DeepSeek model architecture. It modifies the standard Transformer by replacing the core components with Multi-Head Latent Attention (MLA) and a Mixture-of-Experts (MoE) layer. This design also utilizes RMS Norm (Root Mean Square Normalization) and a specialized Decoupled RoPE (Rotary Position Embedding).

Use the architecture diagram to compare the standard Transformer with a language model that substitutes Multi-Head Latent Attention and Mixture-of-Experts. section 1.2.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains pretraining, inference, Transformer components, and efficiency-oriented replacements, but defers complete production-scale training.
Examples
Uses a travel-plan prompt, GPT-style Transformer systems, and a scaled-down DeepSeek implementation as concrete examples.
What you can do after
Build it from scratch by implementing core language-model architecture and training innovations in a scaled-down model.
How it connects
Connects pretraining and inference with Transformers, self-attention, feed-forward networks, Mixture-of-Experts, quantization, and post-training.

Also explained in

Compare how 89 books explain language model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.