liveBook · Manning

ASHA scheduler

As explained in Transformers in Action

The ASHA scheduler is an asynchronous successive-halving algorithm that allocates resources preferentially to promising hyperparameter trials and stops poor trials early.

The ASHA scheduler is an asynchronous successive-halving algorithm for hyperparameter optimization. It allocates resources preferentially to promising trials and early-stops underperforming configurations, reducing tuning time compared with exhaustive searches. section 8.2

Configuring ASHA with Ray Tune

A Ray Tune job searches learning rate, epochs, batch size, and weight decay against eval_loss. ASHA permits one epoch of initial progress, considers up to eight epochs, and uses a reduction factor of two before stopping weak trials.

How Transformers in Action covers it

How deep it goes
Configures maximum training time, grace period, and reduction factor, but does not explain the scheduler's internal algorithm.
Examples
Uses Ray Tune to search learning rate, epochs, batch size, and weight decay while minimizing eval_loss.
What it assumes
Assumes a Ray training workflow, a defined search space, and a reported evaluation metric.
What you can do after
Operate/configure Ray Tune searches with ASHA and select the best trial for full retraining.
How it connects
Connects hyperparameter optimization with distributed training, early stopping, evaluation metrics, and checkpointing.
Cover of Transformers in Action
liveBookmanning.com
Read Transformers in Action on liveBook Get the book at manning.com

Also explained in

Compare how 2 books explain ASHA scheduler →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.