FP4
FP4 is a family of four-bit floating-point formats for representing numerical values at reduced precision.
Quantization and Fast Inference
A family of 4-bit floating-point formats for reduced-precision numerical representation.
- How it is used
- Compresses model weights for low-memory inference while trading numerical resolution against storage and hardware throughput.
- How deep it goes
- Defines E2M1 encoding and compares scaling configurations with measured model quality and memory use.
- Examples
- Compares per-tensor, block-16, and block-32 FP4 inference on OPT-6.7B using WikiText-2 perplexity and estimated model size.
- What you can do after
- Apply blockwise scaling and match FP4 deployment to GPU support to balance model quality, storage, and throughput.
CUDA for LLMs
FP4 is a four-bit floating-point numerical format used for low-precision representation and computation.
- How it is used
- Reduces precision for deep-learning computation, especially inference, to raise tensor-core throughput and ease memory-bandwidth pressure.
- How deep it goes
- Defines FP4 fields and block scaling, then sketches CUTLASS use; defers detailed implementation and calibration complexities.
- Examples
- Uses MXFP4 and NVFP4 block scales, a CUTLASS GEMM configuration, and an 8192³ NVFP4 benchmark.
- What you can do after
- Apply FP4 through CUTLASS to configure and benchmark a Blackwell SM100 GEMM.
What to know first
- bfloat16bfloat16 is a 16-bit floating-point format with one sign bit, eight exponent bits, and seven fraction bits, providing a wide numerical range with reduced precision.
- latencyLatency is the elapsed time between initiating an operation or request and receiving its corresponding result or observable response.
- floating-point operations per secondFloating-point operations per second (FLOP/s) is a measure of computational throughput expressed as the number of floating-point operations performed per second.
- throughputThroughput is the rate at which a system processes or completes work, data, or requests over a specified period.
Related concepts
- tcgen05used by · TCGen05 is Blackwell’s next-generation tensor-core instruction set and programming interface, supporting newer precisions such as FP4.
- int4 quantizationcontrasts with · INT4 quantization represents numerical values using 4-bit integers, reducing storage requirements while limiting numerical precision.
- FP8contrasts with · FP8 is a family of 8-bit floating-point formats for representing values and performing reduced-precision computation.
- float16contrasts with · float16 is a 16-bit floating-point format, commonly IEEE 754 binary16, with less precision and range than 32-bit floating-point formats.
Questions about FP4
What is FP4?
FP4 is a family of four-bit floating-point formats for representing numerical values at reduced precision.
Which Manning books explain FP4?
Quantization and Fast Inference and CUDA for LLMs.
What should I know before learning FP4?
Manning books that teach FP4 usually introduce bfloat16, latency, floating-point operations per second and throughput first.
What is the difference between FP4 and int4 quantization?
FP4 is a family of four-bit floating-point formats for representing numerical values at reduced precision. INT4 quantization represents numerical values using 4-bit integers, reducing storage requirements while limiting numerical precision.
What is FP4 a kind of?
FP4 is a kind of low-bit floating-point format. A low-bit floating-point format is a floating-point representation that uses substantially fewer bits than conventional formats such as FP32.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

