Gradient accumulation

As explained in Building Reliable AI Systems

A technique that sums gradients across multiple mini-batches before updating model parameters.

training_args = TrainingArguments(   #1
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    num_train_epochs=1,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    output_dir="./finetuned-llama2-bitext",
    save_strategy="no"
)

How Building Reliable AI Systems covers it

How deep it goes

Defines the memory-saving mechanism and gives a batch-size-one, four-step configuration.

Examples

Uses a per-device batch size of 1 with accumulation over 4 steps to reach an effective batch size of 4.

What it assumes

Introduces mini-batches, batch size, and parameter updates while defining gradient accumulation.

What you can do after

Apply it through Transformers TrainingArguments by accumulating four steps at a per-device batch size of one.

How it connects

Connects gradient accumulation to mini-batches, batch size, and parameter updates.

liveBookmanning.com

Read Building Reliable AI Systems on liveBook Get the book at manning.com

Related concepts

• batch sizeproduces · Batch size is the number of items grouped together for processing or computation as a single batch.
• gpu video memoryaddresses · GPU memory is memory accessible to a graphics processing unit for storing data and computation state used in graphics or other workloads.
• memory usageaddresses · Memory usage is the amount of computer memory occupied or allocated by a program, process, data, or computation during operation.
• backpropagationuses · Backpropagation is a procedure for computing derivatives of an output or loss with respect to inputs and parameters by applying the chain rule backward through a computational graph.
• data parallelused by · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.

This explainer was generated by Manning's knowledge graph from the text of Building Reliable AI Systems. The section links point to the parts of the book it draws on.