Asynchronous training

As explained in Reinforcement Learning from Human Feedback

Asynchronous reinforcement learning is an RL training arrangement in which rollout generation and policy optimization proceed concurrently on separate workers with periodically synchronized parameters.

Asynchronous training separates rollout generation from policy optimization. Actor systems generate answers while a learner takes policy-gradient steps, so the two sides can run concurrently and may temporarily hold different model weights. This replaces a synchronized generation-update rhythm with a pipeline fed by completed rollouts. section 6.3.3 section 6.6

The arrangement targets a throughput problem in long-horizon reasoning RL: completion lengths can vary sharply, including very long outliers, leaving compute idle while a synchronized batch waits. Asynchronous updates let training consume recently completed rollouts from multiple generating instances instead of waiting for every rollout to finish. section 7.3.3 section 6.3.3

A practical asynchronous pipeline has these responsibilities:

• Actors try to solve problems and emit completed rollouts; learner workers consume them for policy-gradient updates. section 6.3.3
• Queues pass rollout data between actor and learner GPUs, allowing generation and learning to proceed independently. section 6.3.3
• Weight synchronization refreshes actor parameters from the learner; longer intervals can support scaling across multiple data centers. section 6.3.3
• Off-policy correction must address the distribution mismatch created when rollouts were generated under different weights than the learner's current policy. section 6.6

Actor–learner flow

Several actor instances finish reasoning rollouts at different times and place them in a data queue. The learner fills an update batch from the newest completed items, takes gradient steps, then periodically synchronizes weights back to actors. Fast completions need not wait for a long outlier, but the learner must account for the resulting off-policy data. section 6.3.3 section 7.3.3 section 6.6

Figure 6.8 A comparison of the generation-update phases for synchronous or asynchronous RL training, following Noukhovitch et al. (2024)

Compare synchronized generation-update phases with an asynchronous pipeline to see where concurrent work removes waiting. section 6.3.3

Figure 6.9 An example distributed RL system where two queues are managed to pass data to the learner and actor GPUs, which can both be synchronized with a distributed computing library such as Ray. Olmo Team 2025, license CC-BY.

Inspect a distributed layout with rollout queues connecting actor and learner GPUs, including synchronization through a distributed computing library. section 6.3.3

How Reinforcement Learning from Human Feedback covers it

How deep it goes

Explains actor-learner separation, rollout queues, weight synchronization, and distribution mismatch, while deferring broader distributed-infrastructure issues.

Examples

Illustrates synchronized versus asynchronous phases and a queue-based actor–learner GPU system synchronized with Ray.

What you can do after

Evaluate/understand how concurrent rollout generation and learning trade idle time for off-policy correction requirements.

How it connects

Connects policy gradients, off-policy data, importance sampling, reasoning models, and distributed RL systems.

liveBookmanning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

• data driftproduces · Data drift is a change over time in the statistical distribution or characteristics of data relative to a reference distribution or set of characteristics.
• trainingkind of · Training is a process of developing or improving a person’s or model’s capabilities through instruction, experience, or adjustment using data.
• actorhas part · An actor is an independent entity or participant that performs actions, maintains or pursues state or goals, and interacts with other entities or systems.
• work queue patternuses · The work queue pattern stores pending units of work for workers or processors to retrieve and process, often asynchronously.
• trajectoryuses · A trajectory is an ordered sequence of states, actions, events, or operations that records or describes progression through successive steps or over time.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.