liveBook · Manning

Bfloat16

As explained in Rearchitecting LLMs

bfloat16 is a 16-bit floating-point format with a dynamic range similar to float32 and reduced precision and storage requirements.

bfloat16 is a 16-bit floating-point format with a dynamic range similar to Float32. Training configuration can enable it with `bf16=True`; on an L4 GPU, it is supported natively and is described as more stable than Float16 because it avoids overflow by preserving Float32’s dynamic range. section 7.3.3

Listing 7.11 Configuring and training QLoRA
from trl import SFTTrainer, SFTConfig #1
 
training_args = SFTConfig( 
    output_dir="./qlora_output",
    num_train_epochs=3, #2
    per_device_train_batch_size=8,
    gradient_accumulation_steps=1,
    learning_rate=2e-4, #3
    lr_scheduler_type="cosine", #4
    warmup_steps=7, #5
    fp16=False,
    bf16=True, #6
    logging_steps=10,
    save_strategy="no",
    max_length=MAX_SEQ_LENGTH,         
    dataset_text_field="text", #7
    report_to="none",
)
 
trainer = SFTTrainer(
    model=model,
    train_dataset=train_dataset,
    args=training_args,
)
trainer.train()
 

The training configuration sets `bf16=True` alongside `fp16=False`. section 7.3.3

How Rearchitecting LLMs covers it

How deep it goes
Compares dynamic range and overflow behavior with Float32 and Float16, without detailing format internals.
Examples
Shows `bf16=True` and `fp16=False` in an SFTConfig used for training on an L4 GPU.
What you can do after
Configure QLoRA training to use bfloat16 on an L4 GPU that supports it natively.
How it connects
Connects bfloat16 with Float32 dynamic range, Float16 overflow behavior, and QLoRA training.

Also explained in

Compare how 6 books explain bfloat16 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.