ASHA scheduler
The ASHA scheduler is an asynchronous successive-halving algorithm that allocates resources to promising hyperparameter trials while stopping underperforming trials early.
Designing Deep Learning Systems: A guide for software engineers
ASHA is a scalable asynchronous successive-halving algorithm for early stopping of underperforming trials and allocating resources to promising trials.
- How it is used
- Deploys early stopping inside Ray Tune to reduce wasted training during distributed hyperparameter optimization.
- How deep it goes
- Defines high-level trial termination and resource allocation but does not detail the algorithm's internal mechanics.
- Examples
- Uses a ConvNet training objective that reports accuracy while searching learning rate and momentum.
- What you can do after
- Apply it via Ray Tune to run more efficient, parallel hyperparameter searches.
Read how Designing Deep Learning Systems: A guide for software engineers explains ASHA scheduler →
Transformers in Action
The ASHA scheduler is an asynchronous successive-halving algorithm that allocates resources preferentially to promising hyperparameter trials and stops poor trials early.
- How it is used
- Accelerates distributed hyperparameter searches by stopping weak training trials and retaining promising configurations.
- How deep it goes
- Configures maximum training time, grace period, and reduction factor, but does not explain the scheduler's internal algorithm.
- Examples
- Uses Ray Tune to search learning rate, epochs, batch size, and weight decay while minimizing eval_loss.
- What you can do after
- Operate/configure Ray Tune searches with ASHA and select the best trial for full retraining.
Related concepts
- Ray Tuneuses · Ray Tune is a Ray library for distributed hyperparameter tuning and experiment management.
- hyperparameter tuningaddresses · Hyperparameter tuning is the process of selecting or adjusting configurable model or procedure settings and evaluating them against a chosen objective.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.