liveBook · Manning

Marlin kernel

As explained in Quantization and Fast Inference

A Marlin kernel is an optimized CUDA matrix-multiplication implementation for low-bit large-language-model inference.

A Marlin kernel is an optimized CUDA matrix-multiplication kernel for LLM inference. It unpacks packed four-bit AWQ weights and multiplies them in a single pass, avoiding full-precision weight traffic. section 12.1

Hardware support determines whether inference actually uses the four-bit kernel:

  • vLLM's Marlin-family AWQ INT4 kernels support compute capability sm80 (Ampere) and newer; an Ada L4 at sm89 qualifies. section 12.1
  • A Turing T4 at sm75 has no matching kernel; loading may fail or weights may be dequantized to FP16 before matrix multiplication. section 12.1

How Quantization and Fast Inference covers it

How deep it goes
Defines fused unpacking and multiplication, then identifies hardware support and fallback, but does not detail kernel internals.
Examples
Contrasts serving an AWQ model on an Ada L4 (sm89) with a Turing T4 (sm75).
What you can do after
Evaluate whether a GPU can serve AWQ weights in four bits by checking for sm80-or-newer Marlin support.
How it connects
Connects CUDA matrix multiplication, AWQ quantization, INT4 weights, vLLM, and GPU compute capability.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.