liveBook · Manning

Dequantization

As explained in Build a DeepSeek Model (From Scratch)

Dequantization is the transformation of quantized values back into a higher-precision numerical representation using quantization scales.

Dequantization restores the magnitude of a low-precision matrix result by multiplying it with the corresponding activation and weight scaling factors. In fine-grained FP8 computation, Tensor Cores produce the intermediate result, while CUDA Cores perform this conversion before the next framework stage. section 5.6.2

A fine-grained dequantization path proceeds as follows:

  • Partition activations into chunks and weights into blocks, assigning each region its own scaling factor. section 5.6.2
  • Feed the quantized FP8 values to Tensor Cores for the bulk matrix multiplication. section 5.6.2
  • Accumulate partial results in an FP32 CUDA Core register, periodically moving them out of the low-precision Tensor Core environment. section 5.6.3
  • Multiply the accumulated result by the relevant scaling factors and store the restored output as BF16 for the next layer. section 5.6.3

Fine-grained matrix multiplication

For an output term Yᵢⱼ = Wᵢ₁X₁ⱼ + Wᵢ₂X₂ⱼ + ... + WᵢₖXₖⱼ, Tensor Cores calculate low-precision products and partial sums. CUDA Cores then combine those partial sums in FP32 and apply the activation and weight scales, producing a de-quantized value with restored magnitude and improved numerical stability. section 5.6.2 section 5.6.3

Figure 5.29 The complete Fine-Grained Quantization workflow.

Use the complete workflow diagram to trace independent activation and weight scaling through Tensor Core computation and CUDA Core de-quantization. section 5.6.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains scaling-factor multiplication, Tensor Core handoff, FP32 accumulation, and BF16 storage without implementation code.
Examples
Uses independently scaled activation chunks, weight blocks, and periodically promoted matrix-multiplication partial sums.
What it assumes
Assumes quantized activations, weights, scaling factors, Tensor Cores, and CUDA Cores; explains their coordinated execution.
What you can do after
Operate an FP8 computation path that restores output magnitude while retaining high-precision accumulation.
How it connects
Connects fine-grained quantization with Tensor Cores, CUDA Cores, FP32 accumulation, and BF16 output.

Also explained in

Compare how 2 books explain dequantization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.