liveBook · Manning

L4 GPU

As explained in Quantization and Fast Inference

L4 is an NVIDIA data-center GPU designed for accelerated inference and other parallel workloads.

The NVIDIA L4 serves as the server GPU in a controlled inference comparison: Qwen2.5-3B-Instruct runs on the same GPU and concurrency sweep while its weights switch between FP16 and AWQ four-bit. GPTQ four-bit and FP8 are configured but were not tested. section 12.3

How Quantization and Fast Inference covers it

How deep it goes
Supports a controlled format comparison across a concurrency sweep but gives no GPU architecture details or results for untested formats.
Examples
Compares Qwen2.5-3B-Instruct in FP16 and AWQ four-bit on the same L4 across a concurrency sweep.
What you can do after
Evaluate weight formats on a fixed GPU and model setup while holding the concurrency sweep constant.
How it connects
Connects to LLM inference, quantization, FP16, AWQ, and concurrency testing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.