Swish activation function
Swish multiplies the input by the sigmoid of β times that input. section 3.3.1
When β = 1, Swish is exactly SiLU. For other β values, the activation shapes differ, so the names are not interchangeable. section 3.3.1
In SwiGLU, Swish activates the gate projection, whose output is multiplied elementwise by the value projection. Fusing these operations avoids materializing intermediate results in global memory. section 3.3.1
Fused SwiGLU computation
Load gate values x1 and value values x2, compute x1 · sigmoid(x1), multiply by x2, and store the output. section 3.3.1



