A Markov decision process (MDP) models sequential decision making as interaction among states, actions, rewards, and transitions. Reinforcement learning uses this structure to estimate reward-to-go and optimize behavior through trial and error. section 10.4 section 6.2
An MDP-based RL formulation typically identifies:
- States and actions, with a policy π(a|s) selecting actions from states. section 6.2
- Rewards and discounted returns, where future rewards are accumulated recursively. section 6.2
- Value functions, which estimate expected return from a state, and state-visitation distributions, which weight frequently visited states. section 6.2
- Rollouts or trajectories, which estimate objectives from sampled state-action sequences instead of enumerating every state. section 6.2
Token-level language-model RL
In RLHF, a prompt is the starting context, each generated token is an action, and the running prefix is the state. The completion supplies the trajectory and reward; token-level MDP training can then estimate advantages and apply policy updates. section 6.2 section 6.3.2
Equation 6.5
\[\label{eq:policy_objective} J(\theta) \;=\; \sum_{s} d^{\pi_\theta}(s) V^{\pi_\theta}(s).\]The policy objective sums each policy-conditioned state value weighted by the frequency with which the policy visits that state. section 6.2
