A Marlin kernel is an optimized CUDA matrix-multiplication kernel for LLM inference. It unpacks packed four-bit AWQ weights and multiplies them in a single pass, avoiding full-precision weight traffic. section 12.1
Hardware support determines whether inference actually uses the four-bit kernel:
- vLLM's Marlin-family AWQ INT4 kernels support compute capability sm80 (Ampere) and newer; an Ada L4 at sm89 qualifies. section 12.1
- A Turing T4 at sm75 has no matching kernel; loading may fail or weights may be dequantized to FP16 before matrix multiplication. section 12.1
