liveBook · Manning

Compute capability

As explained in Quantization and Fast Inference

Compute capability is a GPU architecture identifier that specifies hardware features relevant to supported operations and kernels.

Compute capability identifies GPU architecture features that determine whether a serving runtime can execute a supported kernel. For AWQ weights, vLLM uses Marlin-family INT4 kernels on compute capability sm80 and newer; without a matching kernel, serving may fail or dequantize weights to FP16. section 12.1 section 12.5

L4 versus T4

An Ada L4 has compute capability sm89, so it runs the four-bit kernel. A Turing T4 has sm75, below the kernel floor, and may fall back to FP16. Quantization and serving therefore have different hardware requirements. section 12.1 section 12.5

How Quantization and Fast Inference covers it

How deep it goes
Defines the sm80 kernel floor and traces its consequences through INT4 execution, load failure, or FP16 fallback.
Examples
Compares an sm89 L4 that runs AWQ at four bits with an sm75 T4 that lacks a matching kernel.
What you can do after
Diagnose whether a GPU can serve AWQ weights with the intended INT4 kernel or will fail or fall back to FP16.
How it connects
Connects GPU architecture identifiers to AWQ, INT4 kernels, and vLLM kernel dispatch.

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.