liveBook · Manning

State

As explained in Build a DeepSeek Model (From Scratch)

A state is the information describing the current situation from which a reinforcement-learning policy selects an action.

A state describes the current situation from which a reinforcement-learning policy selects an action. After the agent acts, the environment changes, returns a reward, and provides a new state. The agent then repeats the cycle. section 7.1 section 7.1.1

A cleaning robot

The state can include where dust remains, where furniture blocks movement, and whether the battery is low. The policy may choose to vacuum or return to the dock; the action changes the room or battery state and can produce a reward or penalty. section 7.1.1

For language generation, the state at step k is the prompt plus all tokens generated so far. The action is the next sampled token, and the transition appends it to the context. A reward function scores the completed response. Sampling during rollouts lets training explore different responses to the same prompt. section 7.1.2

Figure 7.3 Mapping reinforcement learning concepts to LLM generation. The growing context is the state; each sampled next token is an action, and the final response receives a reward.

Follow the growing context as the state changes with each sampled token. section 7.1.2

In a policy-gradient update, πθ(aₖ | sₖ) is the probability assigned to the sampled token aₖ given state sₖ. A simplified update weights the gradient of its log probability by the response reward; practical implementations also handle details such as sequence masks and variable response lengths. section 7.2.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines state transitions and relates state to a simplified policy-gradient update, while noting implementation details omitted from the equation.
Examples
Illustrates state with a cleaning robot's room and a growing response to a tennis-ball arithmetic prompt.
What you can do after
Evaluate how a sampled token changes the state and enters a reward-weighted policy update.
How it connects
Connects state to policy, action, reward, token sampling, and policy gradients.

Also explained in

Compare how 52 books explain state →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.