liveBook · Manning

Precision

As explained in Domain-Specific Small Language Models

A numeric representation setting that determines how many bits are used to store or compute model values during serving.

Precision sets how many bits represent model values. In language model operations, floating-point precision affects memory use and computational speed. section 6.5

Common format tradeoffs include:

  • A 32-bit float uses 1 sign bit, 8 exponent bits, and 23 mantissa bits. section 6.5
  • 16-bit floating point halves memory requirements compared with 32-bit, but can reduce performance. Mixed-precision training switches between 32-bit and 16-bit operations. section 6.5
  • BF16 offers a wider dynamic range than FP16, with less precision. section 6.5

Quantization is a related but distinct approach: it converts floating-point weights to low-bit integers to reduce model size and support faster inference. section 6.0 section 6.5

How Domain-Specific Small Language Models covers it

How deep it goes
Defines 32-bit float bit allocation and contrasts 16-bit, mixed-precision, and BF16 behavior.
Examples
Uses 32-bit float bit allocation and FP16/BF16 comparisons as format examples.
What you can do after
Evaluate memory and speed tradeoffs among floating-point formats and distinguish them from low-bit quantization.
How it connects
Connects floating-point precision to mixed-precision training, BF16, FP16, and quantization.

Also explained in

Compare how 10 books explain precision →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.