A language model learns statistical relationships from vast text datasets and uses fixed parameters to predict the next token, generating human-language sequences during inference. section 2.1.1
Modern language models combine learning phases with Transformer-based computation:
- Pretraining adjusts model weights over vast datasets to learn grammar, facts, reasoning patterns, and relationships between words. section 2.1.1
- Inference uses fixed weights to generate text without further learning. section 2.1.1
- A standard Transformer layer combines multi-head self-attention with a feed-forward neural network. section 1.2.1
- Scaled architectures replace these components with alternatives such as Multi-Head Latent Attention and Mixture-of-Experts to address attention efficiency and model capacity. section 1.2.1
Inference after pretraining
Asking a model to make a travel plan for Italy illustrates inference: the model does not update its parameters, but predicts successive tokens using what pretraining taught it. section 2.1.1
Figure 1.4 A simplified view of the DeepSeek model architecture. It modifies the standard Transformer by replacing the core components with Multi-Head Latent Attention (MLA) and a Mixture-of-Experts (MoE) layer. This design also utilizes RMS Norm (Root Mean Square Normalization) and a specialized Decoupled RoPE (Rotary Position Embedding).

Use the architecture diagram to compare the standard Transformer with a language model that substitutes Multi-Head Latent Attention and Mixture-of-Experts. section 1.2.1
























































































