A retailer with low stock must decide whether to order more now or hold back because demand may fall. A policy is the agent’s decision rule for situations like this: it maps each state to an action, or to probabilities for possible actions. A good policy weighs immediate gains against future consequences. section 2.3 section 10.3
In reinforcement learning, the agent observes a state, chooses an action, receives a reward, and encounters a new state. Experience from these interactions can help it improve its policy. The policy guides behavior; the reward design steers what behavior the agent is encouraged to learn. section 1.7 section 2.4
A policy can take different forms:
- A deterministic policy selects an action for each state. In resource allocation, a state records remaining budget and project outcomes, and an action identifies a project and allocation. section 2.3 section 4.4
- A stochastic policy assigns probabilities to actions. In language generation, the current sequence is the state, and the model samples the next token from a distribution over its vocabulary. section 10.3
To assess a policy, policy evaluation estimates how valuable each state is when the agent follows it. It averages possible outcomes using their probabilities and rewards, repeatedly updating estimates until changes are negligible. Policy improvement then uses those values to look for better actions. section 4.2 section 4.4
Listing 4.6 Policy Evaluation Function
def policy_evaluation(mdp: ResourceAllocationMDP,
states: list[tuple],
policy: dict[tuple, tuple],
value_function: dict[tuple, float],
theta: float = 1e-6) -> dict[tuple, float]:
actions = create_action_space(mdp)
iteration = 0
while True:
delta = 0.0
new_value_function = value_function.copy()
for state in states:
old_value = value_function[state]
action = policy.get(state, (-1, 0)) #A
expected_value = 0.0
for next_state in states:
trans_prob = transition_probability(mdp, state, action, next_state)
if trans_prob > 0:
reward = reward_function(mdp, state, action, next_state)
expected_value += trans_prob * (reward + mdp.discount_factor * value_function[next_state]) #B
new_value_function[state] = expected_value
delta = max(delta, abs(old_value - expected_value)) #C
value_function = new_value_function #D
iteration += 1
if delta < theta: #E
break
print(f"Policy evaluation converged in {iteration} iterations")
return value_function
This implementation evaluates a resource-allocation policy by repeatedly updating state values from possible next states. section 4.4
A policy need not be represented directly. A neural network can represent a policy’s action probabilities, a value function used to choose actions, or both. For example, REINFORCE updates a policy directly, while DQN acts greedily on a learned value function. section 12.1




















