MiniDeepSeek is a compact DeepSeek-style language model that combines shared token embeddings, Transformer blocks, final normalization, vocabulary projection, and a multi-token prediction chain in one PyTorch module. section 6.2.2
Its architecture separates training and inference around different efficiency requirements:
- Inference uses past key-value caches, position offsets, causal masks, and logits for only the final token. section 6.2.2
- Training processes complete sequences without the KV cache, computes standard next-token loss, and adds losses from the multi-token prediction chain. section 6.2.2
The complete system extends beyond model.py: dataset preparation, model initialization, optimization, mixed-precision training, validation, checkpointing, and text generation form a trainable foundation-model pipeline. section 6.0 section 6.3.3
A runnable training configuration
A flagship configuration uses a 768-dimensional model, 12 Transformer layers, 12 attention heads, and roughly 130 million parameters, trained on TinyStories with AdamW and periodic validation checkpoints.
Listing 6.16 Initializing the MiniDeepSeek Model
class MiniDeepSeek(nn.Module):
def __init__(self, args: ModelArgs):
super().__init__()
self.args = args
# Core model components
self.embed = nn.Embedding(args.vocab_size, args.d_model) #A
self.blocks = nn.ModuleList([TransformerBlock(args) for _ in range(args.n_layers)]) #B
self.norm_f = nn.LayerNorm(args.d_model)
self.lm_head = nn.Linear(args.d_model, args.vocab_size, bias=False) #C
# Multi-Token Prediction chain
self.mtp_modules = nn.ModuleList([MTPModule(args) for _ in range(args.n_mtp_modules)]) #D
def causal_mask(self, S: int, device) -> torch.Tensor: #E
mask = torch.triu(torch.ones(S, S, device=device, dtype=torch.bool), diagonal=1)
return torch.zeros(S, S, device=device).masked_fill(mask, float('-inf'))Inspect the top-level PyTorch construction, including embeddings, Transformer blocks, normalization, language-model head, MTP modules, and causal masking. section 6.2.2
