liveBook · Manning

Action

As explained in Build a DeepSeek Model (From Scratch)

An action is a choice made by an agent at a decision point that can change the state of an environment.

An action is a choice made by an agent at a decision point that can change the environment's state. Reinforcement learning repeats the loop: observe a state, choose an action, receive a reward, and observe the updated state. Unlike supervised learning, the agent may receive no exact action target and may learn from delayed feedback. section 7.1 section 7.1.1

The action interface connects four operational objects:

For language generation, the state is the prompt plus tokens already generated, and the action is the next sampled token. Appending that token creates the next state; a completed response receives a reward. Sampling, rather than always taking argmax, preserves exploration across possible solution paths. section 7.1.2 section 7.2.1

A sampled-token trajectory

For the tennis-ball prompt, the model begins with only the prompt as its state. It samples a first token, then samples again from the prompt extended by that token, continuing until a complete answer is produced. The response is a trajectory of states and sampled actions; its reward trains the token probabilities along that trajectory. section 7.1.2 section 7.2.1

Figure 7.2 The agent-environment interface. The cleaning robot observes the room, chooses an action, receives a reward, and then repeats the cycle from the updated room state.

Trace the repeating agent-environment cycle from observation through action and reward to the updated state. section 7.1.1

Figure 7.3 Mapping reinforcement learning concepts to LLM generation. The growing context is the state; each sampled next token is an action, and the final response receives a reward.

Map states, sampled token actions, transitions, and final response rewards onto language generation. section 7.1.2

θₜ₊₁ = θₜ + α ∑ₖ ∇θ log πθ(aₖ | sₖ) R

Use the policy-gradient equation to connect sampled-action log probabilities and reward to parameter updates. section 7.2.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains action through state transitions, stochastic sampling, trajectories, and a basic policy-gradient equation, while deferring implementation details.
Examples
Uses a cleaning robot, sampled language-model tokens, and a tennis-ball arithmetic response as concrete action examples.
What it assumes
Assumes basic model-generated tokens but grounds reinforcement learning through states, actions, rewards, policies, and interaction loops.
What you can do after
Apply action sampling to construct language-model rollouts and relate sampled-token probabilities to policy-gradient updates.
How it connects
Connects action to state, reward, policy, transition, trajectory, exploration, and policy-gradient training.

Also explained in

Compare how 17 books explain action →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.