A trajectory distribution assigns probability to complete interaction sequences generated by an initial-state distribution, a policy, and environment dynamics. For a trajectory τ of states, actions, and rewards, its probability multiplies the initial-state probability, each policy action probability, and each environment transition probability. section 3.1
The factorization separates the sources of randomness:
- The initial-state distribution selects s₀. section 3.1
- The policy π(aₜ|sₜ) selects actions from the current state. section 3.1
- The transition dynamics p(sₜ₊₁|sₜ,aₜ) generate subsequent states. section 3.1
- Only the policy term depends on policy parameters θ, which makes the trajectory distribution useful for policy-gradient derivations. section 6.2.1
From trajectories to policy gradients
Writing the objective as an expectation over τ makes it estimable with Monte Carlo rollouts. Applying the log-derivative trick yields an update weighted by trajectory reward or a per-step advantage: ∇θJ(θ)=E[Σₜ∇θlogπθ(aₜ|sₜ)Ψₜ]. A corresponding implementation sums masked token log-probabilities and multiplies them by advantages before backpropagation. section 6.2.1
Equation 3.1
\[\label{eq:rl_dynam} {p_{\pi}(\tau)=\rho_0(s_0)\prod_{t=0}^{T-1}\pi(a_t\mid s_t)p(s_{t+1}\mid s_t,a_t).}\]Inspect the finite-horizon factorization of the trajectory distribution into initial-state, policy, and transition terms. section 3.1
