An action is a choice made by an agent at a decision point that can change the environment's state. Reinforcement learning repeats the loop: observe a state, choose an action, receive a reward, and observe the updated state. Unlike supervised learning, the agent may receive no exact action target and may learn from delayed feedback. section 7.1 section 7.1.1
The action interface connects four operational objects:
- State: the information available before choosing. section 7.1.1 section 7.1.2
- Action: the selected operation that produces a transition. section 7.1.1 section 7.1.2
- Reward: feedback used to favor actions associated with better outcomes. section 7.1.1 section 7.2.2
- Policy: the parameterized rule assigning action probabilities to states. section 7.1.1 section 7.2.2
For language generation, the state is the prompt plus tokens already generated, and the action is the next sampled token. Appending that token creates the next state; a completed response receives a reward. Sampling, rather than always taking argmax, preserves exploration across possible solution paths. section 7.1.2 section 7.2.1
A sampled-token trajectory
For the tennis-ball prompt, the model begins with only the prompt as its state. It samples a first token, then samples again from the prompt extended by that token, continuing until a complete answer is produced. The response is a trajectory of states and sampled actions; its reward trains the token probabilities along that trajectory. section 7.1.2 section 7.2.1
Figure 7.2 The agent-environment interface. The cleaning robot observes the room, chooses an action, receives a reward, and then repeats the cycle from the updated room state.

Trace the repeating agent-environment cycle from observation through action and reward to the updated state. section 7.1.1
Figure 7.3 Mapping reinforcement learning concepts to LLM generation. The growing context is the state; each sampled next token is an action, and the final response receives a reward.

Map states, sampled token actions, transitions, and final response rewards onto language generation. section 7.1.2
Use the policy-gradient equation to connect sampled-action log probabilities and reward to parameter updates. section 7.2.2
















