appendix-a

Appendix A. A Gemma-3 case study in SLM architectural refinements

 

In Chapter 3 we explored the main architectural optimizations popularized by Meta's Llama Model family: Grouped-Query Attention (GQA) that optimizes memory consumption and the Gated Linear Unit (GLU) that enables more sophisticated Multi-Layer Perceptron (MLP) processing with enhanced contextualization capabilities.

Almost every new model family adopts these two optimizations and adds its own refinements.

Model sizes now vary widely. Optimization strategies that work in models with hundreds of billions of parameters aren't always applicable to smaller models with just a few hundred million.

Designing an SLM involves architectural decisions and trade-offs that prioritize compactness and inference speed. We’ll study the architecture of Google’s Gemma-3-270M, a model that shows how the modern standard is refined for small models.

Let's load the model:

model = AutoModelForCausalLM.from_pretrained(
   "google/gemma-3-270m",
    torch_dtype="auto", 
    device_map="cpu",
    trust_remote_code=True
)

And let's examine its structure in the following listing:

A.1 Attention in SLMs from GQA to MQA

A.2 Refinements in the MLP module: expansion and stability