bfloat16 is a 16-bit floating-point format with a dynamic range similar to Float32. Training configuration can enable it with `bf16=True`; on an L4 GPU, it is supported natively and is described as more stable than Float16 because it avoids overflow by preserving Float32’s dynamic range. section 7.3.3
Listing 7.11 Configuring and training QLoRA
from trl import SFTTrainer, SFTConfig #1
training_args = SFTConfig(
output_dir="./qlora_output",
num_train_epochs=3, #2
per_device_train_batch_size=8,
gradient_accumulation_steps=1,
learning_rate=2e-4, #3
lr_scheduler_type="cosine", #4
warmup_steps=7, #5
fp16=False,
bf16=True, #6
logging_steps=10,
save_strategy="no",
max_length=MAX_SEQ_LENGTH,
dataset_text_field="text", #7
report_to="none",
)
trainer = SFTTrainer(
model=model,
train_dataset=train_dataset,
args=training_args,
)
trainer.train()
The training configuration sets `bf16=True` alongside `fp16=False`. section 7.3.3





