Llama is Meta’s family of autoregressive large language models. Its importance comes less from winning every benchmark than from making capable model experimentation and production tooling widely accessible. section 4.1.3 section 9.1
Model size creates an immediate operational trade-off: larger variants can represent more information but demand substantially more GPU memory. Llama variants span 7B through 65B parameters, while Llama 2 offers 7B, 13B, and 70B models. section 3.4.7 section 4.1.3
Key characteristics include:
- Llama 2 used 2 trillion tokens from a curated internet dataset, then applied reinforcement learning from human feedback for chat and code variants. section 4.1.3
- Llama 3’s practical advantage is availability: its surrounding community has produced tooling, optimizations, and additional data work. section 9.1
- A simplified Llama architecture combines RMS normalization, masked multihead attention with RoPE, SwiGLU activation, residual blocks, embeddings, and multiple layers. section 9.2
- Production work commonly requires quantization for serving and LoRA or QLoRA for smaller task-specific updates. section 9.0 section 9.5
A deliberately small Llama project
A PyTorch implementation can define a SimpleLlama model, train it with AdamW and a cosine learning-rate scheduler, and generate short text. The demonstration produces weak, incoherent output because it uses limited data and compute; its goal is to expose the production path rather than reproduce Meta’s quality. section 9.1
Listing 9.4 Simple Llama
class LlamaBlock(nn.Module):
def __init__(self, config):
super().__init__()
self.config = config
self.rms = RMSNormalization(
(config["d_model"], config["d_model"])
) #1
self.attention = RoPEMaskedMultiheadAttention(config).to(device)
self.feedforward = nn.Sequential(
nn.Linear(config["d_model"], config["hidden_dim"]),
SwiGLU(config["hidden_dim"]),
nn.Linear(config["hidden_dim"], config["d_model"]),
)
Inspect the core LlamaBlock definition, including RMS normalization, RoPE-masked attention, feed-forward layers, and SwiGLU activation. section 9.2
