A policy is an agent’s decision rule: it maps a state or observation to an action. In reinforcement learning, the agent tries actions, receives rewards, and refines that rule to increase expected cumulative reward. A self-driving car learning when to merge into traffic illustrates the goal: favor successful merges over dangerous maneuvers. section 12.1.2 section 11.2
Three distinctions help specify what a policy must do:
- A deterministic policy selects an action; a stochastic policy gives probabilities for selecting actions. A parameterized policy can change its behavior when its parameters change. section 12.1.2
- A state contains complete environment information; an observation may contain only part of it. The decision rule must work with what the agent can observe. section 12.1.2
- In an MDP, actions affect future states. Learning a policy therefore means looking beyond the immediate reward toward expected long-term reward. section 12.2 section 12.7
A policy for CartPole
A cart can move left or right to keep a pole upright. Its observation has four values: cart position and velocity, and pole angle and angular velocity. Each time step with the pole upright earns +1. A useful policy selects moves that keep earning rewards, rather than treating either direction as always correct. section 12.3
One way to learn that policy is advantage actor-critic (A2C). The actor turns the current state into action probabilities; the critic estimates how good the selected action is relative to the state’s expected value. After sampling an action and observing the next reward and state, update the critic’s estimate and adjust the actor to favor actions with higher estimated advantage. Track episode returns to see whether the resulting decisions improve. section 12.1.4 section 12.3
class Actor(nn.Module):
def __init__(self, state_dim, action_dim):
super(Actor, self).__init__()
self.fc1 = nn.Linear(state_dim, 64)
self.fc2 = nn.Linear(64, 32)
self.fc3 = nn.Linear(32, action_dim)
def forward(self, state):
x1 = F.relu(self.fc1(state))
x2 = F.relu(self.fc2(x1))
action_probs = F.softmax(self.fc3(x2), dim=-1)
return action_probs
The PyTorch actor shows a policy in executable form: a neural network takes a state and returns action probabilities through softmax. section 12.3
For a library-based route, train an `MlpPolicy` on `CartPole-v1` with Stable-Baselines3 A2C or PPO, then use `evaluate_policy` and `model.predict` to test it. PPO limits the size of policy updates with a clipped objective to promote stability. The same state-to-action framing can be applied to routing or dispatch, but the environment, actions, and reward must fit the problem. section 12.3 section 12.7 section 12.2
