Mixed precision

As explained in Domain-Specific Small Language Models

Using reduced-precision numeric formats for some computations or weights to reduce memory use, often with limited loss in performance.

Mixed precision combines lower-precision and higher-precision numeric formats during training or inference. It reduces memory use and can improve speed without the performance degradation of pure FP16. section 6.1

Operationally, mixed precision uses different formats for different parts of the workload:

• During training, weights and computations can use FP16, gradients can be converted back to FP32 for numerical stability, and scaled gradients update the original weights. section 6.1
• During inference, FP16 plus FP32 uses more memory than pure FP16 but less than pure FP32, with comparable and sometimes better performance. section 6.1
• Lower precision helps models fit hardware constraints: FP16 uses about half the FP32 memory, while INT8 reduces it further. section 4.2

Inference sizing

A 2.7-billion-parameter GPT-Neo model requires about 12.96 GB in FP32 and 5.4 GB in FP16. Mistral 7B requires about 33.6 GB in FP32 or 16.2 GB in FP16. section 4.2

Figure 6.2 Representation of 16-bit floating point numbers

Use the FP16 representation diagram to compare its sign, exponent, and mantissa allocation with FP32. section 6.1

How Domain-Specific Small Language Models covers it

How deep it goes

Explains FP16 and FP32 representation, training-time switching, gradient stability, and inference tradeoffs without detailing hardware kernels.

Examples

Uses GPT-Neo and Mistral 7B memory estimates, FP16 representation diagrams, and BF16 hardware-support checks.

What you can do after

Apply it by selecting lower-precision model formats to meet GPU memory constraints while preserving comparable performance.

How it connects

Connects floating-point precision with model training, inference, BF16, FP16, FP32, and quantization.

liveBookmanning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with R, Third EditionMixed-precision training is model training that uses numerical values with different floating-point precisions to balance computational efficiency and numerical accuracy.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA training technique that uses lower-precision numeric types for some computations while keeping others in higher precision to balance speed, memory use, and numerical stability.

Compare how 3 books explain mixed precision →

Related concepts

• float16uses · Float16 is a 16-bit floating-point numeric format that represents values with less precision and storage than single-precision formats such as float32.
• FP32uses · FP32 is the IEEE 754 binary32 single-precision floating-point format, which represents approximate real numbers using 32 bits.
• PyTorchused by · PyTorch is an open-source machine-learning framework for Python that provides tensor computation, automatic differentiation, and tools for building and training neural networks.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.