The NVIDIA L4 serves as the server GPU in a controlled inference comparison: Qwen2.5-3B-Instruct runs on the same GPU and concurrency sweep while its weights switch between FP16 and AWQ four-bit. GPTQ four-bit and FP8 are configured but were not tested. section 12.3
L4 GPU
L4 is an NVIDIA data-center GPU designed for accelerated inference and other parallel workloads.
How Quantization and Fast Inference covers it
- How deep it goes
- Supports a controlled format comparison across a concurrency sweep but gives no GPU architecture details or results for untested formats.
- Examples
- Compares Qwen2.5-3B-Instruct in FP16 and AWQ four-bit on the same L4 across a concurrency sweep.
- What you can do after
- Evaluate weight formats on a fixed GPU and model setup while holding the concurrency sweep constant.
- How it connects
- Connects to LLM inference, quantization, FP16, AWQ, and concurrency testing.
Related concepts
- GPUkind of · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and other data-parallel workloads.
- bfloat16uses · bfloat16 is a 16-bit floating-point format with one sign bit, eight exponent bits, and seven fraction bits, providing a wide numerical range with reduced precision.
This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.
