Paged optimizer

As explained in Transformers in Action

A paged optimizer is an optimizer implementation that manages optimizer states in pages to reduce memory spikes and support movement between CPU and GPU memory.

A paged optimizer manages optimizer memory in pages during fine-tuning. It addresses memory spikes, especially those caused by gradient checkpointing, by dynamically moving data between CPU and GPU memory. section 8.3.4

Within QLoRA, paging complements low-bit model storage, double quantization, and low-memory adaptation. Instead of requiring all optimizer state to remain on the GPU, the implementation allocates and transfers memory as training demands change, helping avoid out-of-memory failures under tight hardware limits. section 8.3.4 section 8.3.5

Paged optimizers are useful when:

• Gradient checkpointing creates temporary memory spikes that can exceed available GPU capacity. section 8.3.4
• Fine-tuning must run with the minimum practical number of GPUs. section 8.3.5
• Optimizer data must move efficiently between CPU and GPU memory while training continues. section 8.3.4

Figure 8.14 Overview of fine-tuning methods: on the left is the usual fine-tuning, without any optimization; in the middle, LoRA; and on the right, QLoRA. Image is taken from [6].

Compare ordinary fine-tuning, LoRA, and QLoRA to see where paged optimization fits within progressively more memory-efficient adaptation strategies. section 8.3.4

How Transformers in Action covers it

How deep it goes

Explains the paging rationale and CPU/GPU movement but stops short of optimizer algorithms, page layouts, or implementation code.

Examples

Uses QLoRA’s fine-tuning workflow and contrasts ordinary fine-tuning, LoRA, and QLoRA through a comparative figure.

What it assumes

Assumes familiarity with LLM fine-tuning, GPU memory, gradient checkpointing, and quantization.

What you can do after

Evaluate how paged memory management reduces out-of-memory risk during quantized LLM fine-tuning.

How it connects

Connects paged optimizers to QLoRA, quantization, double quantization, gradient checkpointing, and LoRA.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Related concepts

• QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that quantizes a pretrained model’s weights while training low-rank adapters to reduce memory requirements.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.