liveBook · Manning

MiniDeepSeek

As explained in Build a DeepSeek Model (From Scratch)

MiniDeepSeek is the book's compact DeepSeek-style language model that integrates the described architecture components into a single PyTorch module.

MiniDeepSeek is a compact DeepSeek-style language model that combines shared token embeddings, Transformer blocks, final normalization, vocabulary projection, and a multi-token prediction chain in one PyTorch module. section 6.2.2

Its architecture separates training and inference around different efficiency requirements:

  • Inference uses past key-value caches, position offsets, causal masks, and logits for only the final token. section 6.2.2
  • Training processes complete sequences without the KV cache, computes standard next-token loss, and adds losses from the multi-token prediction chain. section 6.2.2

The complete system extends beyond model.py: dataset preparation, model initialization, optimization, mixed-precision training, validation, checkpointing, and text generation form a trainable foundation-model pipeline. section 6.0 section 6.3.3

A runnable training configuration

A flagship configuration uses a 768-dimensional model, 12 Transformer layers, 12 attention heads, and roughly 130 million parameters, trained on TinyStories with AdamW and periodic validation checkpoints.

Listing 6.16 Initializing the MiniDeepSeek Model
class MiniDeepSeek(nn.Module):
    def __init__(self, args: ModelArgs):
        super().__init__()
        self.args = args
        # Core model components
        self.embed = nn.Embedding(args.vocab_size, args.d_model) #A
        self.blocks = nn.ModuleList([TransformerBlock(args) for _ in range(args.n_layers)]) #B
        self.norm_f = nn.LayerNorm(args.d_model)
        self.lm_head = nn.Linear(args.d_model, args.vocab_size, bias=False) #C
        # Multi-Token Prediction chain
        self.mtp_modules = nn.ModuleList([MTPModule(args) for _ in range(args.n_mtp_modules)]) #D
    def causal_mask(self, S: int, device) -> torch.Tensor:  #E
       mask = torch.triu(torch.ones(S, S, device=device, dtype=torch.bool), diagonal=1)
       return torch.zeros(S, S, device=device).masked_fill(mask, float('-inf'))

Inspect the top-level PyTorch construction, including embeddings, Transformer blocks, normalization, language-model head, MTP modules, and causal masking. section 6.2.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Descends from module composition through training and inference control flow, optimization, evaluation, and checkpointing.
Examples
Uses PyTorch listings, a roughly 130-million-parameter configuration, TinyStories batches, validation runs, and saved checkpoints.
What it assumes
Assumes familiarity with Transformer components and PyTorch while grounding their integration in a complete training pipeline.
What you can do after
Build it from scratch, train it on prepared data, evaluate checkpoints, and generate text from the resulting model.
How it connects
Connects Transformer blocks, KV caching, Multi-Head Latent Attention, Mixture-of-Experts, Multi-Token Prediction, FP8, and DualPipe Parallelism.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.