Scalability means increasing model size, training data, or compute in pursuit of better machine-learning performance. The practical question is whether the improvement transfers beyond the measured task. section 19.2
Scaling laws describe a predictable relationship between model or dataset size and performance on specific tasks. However, benchmark gains may mainly reflect memorization, while novelty, phrasing sensitivity, and generalizable reasoning remain unresolved. section 19.2
Keep these practical distinctions in mind:
- Scale input features independently during preprocessing when their value ranges differ. section 4.3.6
- Prefer a small model with one or two intermediate layers when training data is scarce, reducing severe overfitting. section 4.3.6
- Treat larger models and datasets as an empirical strategy, not as proof that a different capability or architecture has emerged. section 19.2
- Scale dot-product attention scores by the square root of vector length to improve softmax-gradient stability. section 15.3.1
Scaling attention scores
In multi-head attention, compute query–key dot products and apply softmax to scores divided by sqrt(head_dim). Longer vectors can produce larger dot products; the scaling factor keeps the softmax gradients more stable. section 15.3.1
query_dense = [layers.Dense(head_dim) for i in range(num_heads)] key_dense = [layers.Dense(head_dim) for i in range(num_heads)] value_dense = [layers.Dense(head_dim) for i in range(num_heads)] output_dense = layers.Dense(head_dim * num_heads) def multi_head_attention(query, key, value): head_outputs = [] for i in range(num_heads): query = query_dense[i](query) key = key_dense[i](key) value = value_dense[i](value) scores = np.einsum("btd,bsd->bts", target, source) scores = softmax(scores / math.sqrt(head_dim), axis=-1) head_output = np.einsum("bts,bsd->btd", scores, source) head_outputs.append(head_output) outputs = ops.concatenate(head_outputs, axis=-1) return output_dense(outputs) multi_head_attention(query=target, key=source, value=source)
Inspect the multi-head attention listing to see scaling applied directly inside the softmax computation. section 15.3.1
