Appendix A. A Gemma-3 case study in SLM architectural refinements
In Chapter 3 we explored the main architectural optimizations popularized by Meta's Llama Model family: Grouped-Query Attention (GQA) that optimizes memory consumption and the Gated Linear Unit (GLU) that enables more sophisticated Multi-Layer Perceptron (MLP) processing with enhanced contextualization capabilities.
But just because a standard has been established doesn't mean innovation has stopped. In fact, the situation is the opposite: modifications happen with almost every new model family, and although they've adopted this fundamental pattern, they've introduced their own refinements.
Additionally, it's important to consider the growing diversity in LLM sizes. Optimization strategies that work in models with hundreds of billions of parameters aren't always applicable to smaller models with just a few hundred million.
One of the most important points is that the optimizations we see don't alter the basic form of the structure, which is especially important because it allows us to apply the same rearchitecting techniques regardless of them.