A trajectory is the complete sequence of states, actions, and rewards experienced by an agent during a finite episode: τ = (s₀, a₀, r₀, s₁, a₁, r₁, …, sₜ, aₜ, rₜ). An initial-state distribution, policy, and environment dynamics jointly induce its probability. section 3.1
A trajectory records the interaction loop and supports optimization:
- The policy selects an action from the current state, π(aₜ|sₜ). section 3.1
- The environment produces a reward and transitions to the next state according to p(sₜ₊₁|sₜ,aₜ). section 3.1
- The agent evaluates accumulated discounted rewards through the return and updates the policy toward higher expected return. section 6.2
- In RLHF, one prompt and one completion form a single-step trajectory or rollout, usually receiving response-level reward with γ = 1. section 3.1.3
CartPole rollout
A CartPole trajectory contains cart position and velocity, pole angle and angular velocity, left or right force actions, per-step rewards, and physics-based transitions. The episode ends when the pole or cart violates its bounds; repeated trajectories let the policy learn corrective pushes that keep the pole balanced longer. section 3.1.2
Equation 3.1
\[\label{eq:rl_dynam} {p_{\pi}(\tau)=\rho_0(s_0)\prod_{t=0}^{T-1}\pi(a_t\mid s_t)p(s_{t+1}\mid s_t,a_t).}\]The trajectory distribution factors into the initial state probability, every policy action probability, and every environment transition. section 3.1
