liveBook · Manning

Policy

As explained in Applied Reinforcement Learning

A policy is a decision rule, possibly probabilistic, that assigns actions or action probabilities to states for an agent.

A retailer with low stock must decide whether to order more now or hold back because demand may fall. A policy is the agent’s decision rule for situations like this: it maps each state to an action, or to probabilities for possible actions. A good policy weighs immediate gains against future consequences. section 2.3 section 10.3

In reinforcement learning, the agent observes a state, chooses an action, receives a reward, and encounters a new state. Experience from these interactions can help it improve its policy. The policy guides behavior; the reward design steers what behavior the agent is encouraged to learn. section 1.7 section 2.4

A policy can take different forms:

  • A deterministic policy selects an action for each state. In resource allocation, a state records remaining budget and project outcomes, and an action identifies a project and allocation. section 2.3 section 4.4
  • A stochastic policy assigns probabilities to actions. In language generation, the current sequence is the state, and the model samples the next token from a distribution over its vocabulary. section 10.3

To assess a policy, policy evaluation estimates how valuable each state is when the agent follows it. It averages possible outcomes using their probabilities and rewards, repeatedly updating estimates until changes are negligible. Policy improvement then uses those values to look for better actions. section 4.2 section 4.4

Listing 4.6 Policy Evaluation Function
def policy_evaluation(mdp: ResourceAllocationMDP, 
                     states: list[tuple], 
                     policy: dict[tuple, tuple], 
                     value_function: dict[tuple, float],
                     theta: float = 1e-6) -> dict[tuple, float]:
 
    actions = create_action_space(mdp)
    
    iteration = 0
    while True:
        delta = 0.0
        new_value_function = value_function.copy()
        
        for state in states:
            old_value = value_function[state]
            
            action = policy.get(state, (-1, 0))  #A
            
            expected_value = 0.0
            for next_state in states:
                trans_prob = transition_probability(mdp, state, action, next_state)
                if trans_prob > 0:
                    reward = reward_function(mdp, state, action, next_state)
                    expected_value += trans_prob * (reward + mdp.discount_factor * value_function[next_state]) #B
            
            new_value_function[state] = expected_value
            delta = max(delta, abs(old_value - expected_value)) #C
        
        value_function = new_value_function #D
        iteration += 1
        
        if delta < theta: #E
            break
    
    print(f"Policy evaluation converged in {iteration} iterations")
    return value_function

This implementation evaluates a resource-allocation policy by repeatedly updating state values from possible next states. section 4.4

A policy need not be represented directly. A neural network can represent a policy’s action probabilities, a value function used to choose actions, or both. For example, REINFORCE updates a policy directly, while DQN acts greedily on a learned value function. section 12.1

How Applied Reinforcement Learning covers it

How deep it goes
Explains deterministic and stochastic policies, then works through iterative evaluation and improvement for resource allocation.
Examples
Illustrates policies with retail inventory, project resource allocation, and language-model token selection.
What you can do after
Builds and evaluates a resource-allocation policy from scratch using dynamic programming.
How it connects
Connects policies to Markov decision processes, reward design, Bellman equations, value functions, and dynamic programming.

Also explained in

Compare how 21 books explain policy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.