Epilogue fusion combines operations after a primary computation in one kernel, keeping intermediate values out of global memory. In SwiGLU, the kernel applies SiLU to the gate projection and multiplies by the value projection, with both operations performed in registers. This avoids intermediate writes and reads that can make the elementwise work memory-bound. section 3.3.1
Simplified SwiGLU
The simplified formula shows the gate activation and value multiplication combined in the fused epilogue. section 3.3.1
Fused SwiGLU in Triton
The kernel splits a contiguous input into gate and value halves, loads both, computes SiLU and multiplies in registers, then stores the output. section 3.3.1
