How deeply does Domain-Specific Small Language Models cover bfloat16?

Defines the bit layout, compares BF16 with FP16, and notes hardware checks without detailing numerical algorithms.

Bfloat16

As explained in Domain-Specific Small Language Models

A 16-bit floating-point format with a wider exponent range and reduced mantissa precision than FP16, used in machine learning.

bfloat16 (BF16) is a 16-bit floating-point format that uses 1 sign bit, 8 exponent bits, and 7 mantissa bits. It extends FP16’s dynamic range while reducing precision. section 6.1

BF16 reduces memory use and generally preserves deep-learning performance, although lower precision can affect some calculations. Models trained in BF16 typically match FP16 mixed precision for inference performance and memory use. section 6.1 section 6.5

Practical considerations:

Use BF16 on supported Google TPUs or NVIDIA Ampere GPUs; check NVIDIA support with torch.cuda.is_bf16_supported(). section 6.1
Prefer BF16 when wider dynamic range matters more than FP16’s additional mantissa precision. section 6.1

How Domain-Specific Small Language Models covers it

Cover of Domain-Specific Small Language Models

liveBook manning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Deep Learning with JAXA 16-bit floating-point format with a float32-like exponent range and reduced significand precision, commonly used for deep learning.
liveBook manning.com

Deep Learning with R, Third Editionbfloat16 is a 16-bit floating-point data type with an 8-bit exponent and a 7-bit mantissa, providing greater numerical range but lower precision than float16.

Compare how 3 books explain bfloat16 →

Related concepts

float16contrasts with · Float16 is a 16-bit floating-point numeric format that represents values with less precision and storage than single-precision formats such as float32.
mixed precisionused by · Mixed precision is a computational technique that uses multiple numerical precisions for different values or operations to balance resource efficiency and numerical accuracy or stability.
QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that quantizes a pretrained model’s weights while training low-rank adapters to reduce memory requirements.
FP32uses · FP32 is the IEEE 754 binary32 single-precision floating-point format, which represents approximate real numbers using 32 bits.
language modelconfigures · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.