A state describes the current situation from which a reinforcement-learning policy selects an action. After the agent acts, the environment changes, returns a reward, and provides a new state. The agent then repeats the cycle. section 7.1 section 7.1.1
A cleaning robot
The state can include where dust remains, where furniture blocks movement, and whether the battery is low. The policy may choose to vacuum or return to the dock; the action changes the room or battery state and can produce a reward or penalty. section 7.1.1
For language generation, the state at step k is the prompt plus all tokens generated so far. The action is the next sampled token, and the transition appends it to the context. A reward function scores the completed response. Sampling during rollouts lets training explore different responses to the same prompt. section 7.1.2
Figure 7.3 Mapping reinforcement learning concepts to LLM generation. The growing context is the state; each sampled next token is an action, and the final response receives a reward.

Follow the growing context as the state changes with each sampled token. section 7.1.2
In a policy-gradient update, πθ(aₖ | sₖ) is the probability assigned to the sampled token aₖ given state sₖ. A simplified update weights the gradient of its log probability by the response reward; practical implementations also handle details such as sequence masks and variable response lengths. section 7.2.2



















































