liveBook · Manning

Bfloat16

As explained in Build a DeepSeek Model (From Scratch)

BF16 is a 16-bit floating-point numerical format with a wide exponent range commonly used for machine-learning computation.

bfloat16 (BF16) is a 16-bit floating-point format with 1 sign bit, 8 exponent bits, and 7 mantissa bits. Matching FP32's 8-bit exponent gives BF16 a wide dynamic range, while the shorter mantissa reduces memory use and precision. section 5.5.3

Use BF16 when range matters more than fine-grained numerical precision:

  • Compared with FP16, BF16 is more resistant to overflow because it preserves FP32's exponent width, although its precision is lower. section 5.5.3
  • Compared with FP32, BF16 halves storage from 32 to 16 bits while retaining the same exponent width. section 5.5.3
  • Compared with FP8 or INT8, BF16 uses more memory but provides substantially more numerical precision or dynamic range. section 5.5.3

Mixed-precision training commonly stores activations in BF16 while converting inputs and weights to FP8 for matrix multiplication. Products accumulate in FP32, results return to BF16, and master weights and updates remain in FP32 for stability. section 5.6.1

KV-cache memory

A BF16 key-value cache uses 2 bytes per parameter. For layers l, batch size b, heads n, head size h, and sequence length s, its size is l × b × n × h × s × 2 × 2: one factor of 2 stores keys and values, and the other represents BF16's two bytes. section 2.5.1

Figure 5.19 A comparison of FP32 and BFloat16.

Visualizes BF16's shared exponent width with FP32 and reduced mantissa width. section 5.5.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines the bit layout, compares numerical trade-offs, and traces BF16 through linear-layer computation and KV-cache sizing.
Examples
Uses bit-layout comparisons, FP8 linear-layer flows, and a KV-cache formula involving BF16's two-byte storage.
What it assumes
Assumes floating-point and neural-network computation concepts while grounding exponent, mantissa, overflow, underflow, and dtype storage.
What you can do after
Evaluate memory, range, and precision trade-offs when selecting BF16 for training, inference storage, or cache design.
How it connects
Connects BF16 with FP32, FP16, FP8, INT8, quantization, mixed precision, linear layers, and KV caches.

Also explained in

Compare how 6 books explain bfloat16 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.