A value function estimates how much future reward an agent can expect from a state or state–action pair when it follows a specified policy. It helps distinguish a choice that pays off immediately from one that leads to better outcomes later. The estimate depends on the policy: changing what the agent does next can change the value. section 12.1.2
Two forms answer different questions:
- The state-value function, Vπ(s), estimates expected cumulative future reward starting in state s and following policy π. It expresses how desirable the state is under that policy. section 12.1.2
- The action-value function, Q(s,a), estimates expected cumulative future reward after taking action a in state s, then following the policy. It evaluates a particular choice rather than the state alone. section 12.1.2
Future rewards can be discounted by γ. A value near 0 emphasizes immediate rewards; a value near 1 gives later rewards more weight. Value functions therefore combine possible future outcomes with a choice about how strongly to value their timing. section 12.1.2
A small decision, then CartPole
In a treasure-seeking example, an action policy assigns 40% probability to moving left and 60% to moving right; weighting the shown possible rewards by their probabilities gives an expected return of 5.728. For CartPole, the same idea becomes a learned estimate: a critic network takes the current observation and predicts a value while an actor chooses between pushing left and pushing right. section 12.1.2 section 12.3
In advantage actor-critic (A2C), the critic provides a value baseline for judging the actor’s action. The implementation forms an advantage from the immediate reward plus the discounted next-state value, minus the current-state value; it omits the next-state value when the episode ends. The critic is updated from squared advantage, while the actor uses the advantage to favor actions judged better than the baseline. section 12.1.4 section 12.3
class Critic(nn.Module):
def __init__(self, state_dim):
super(Critic, self).__init__()
self.fc1 = nn.Linear(state_dim, 64)
self.fc2 = nn.Linear(64, 32)
self.fc3 = nn.Linear(32, 1)
def forward(self, state):
x1 = F.relu(self.fc1(state))
x2 = F.relu(self.fc2(x1))
value = self.fc3(x2)
return value
The PyTorch critic maps a state through fully connected layers to one predicted value, making the value estimate concrete. section 12.3
For the CartPole implementation, γ is set to 0.99, and training tracks critic loss alongside actor loss and episode return. Inspect those measures when experimenting: the value estimate is learned jointly with the policy, not supplied as a known answer. section 12.3 section 12.1.4
