An actor network is the policy network in reinforcement learning: it converts an observed state into action probabilities or action choices. section 12.3
In an actor–critic setup, the actor selects actions while a separate critic estimates state value. A PyTorch actor can use fully connected layers, ReLU activations, and a final softmax over actions. section 12.3
A typical actor interaction and update proceeds as follows:
- Pass the current state through the actor to obtain action probabilities. section 12.3
- Sample an action from a categorical distribution, step the environment, and compute an advantage using the critic. section 12.3
- Update the actor with the sampled action’s log probability weighted by the detached advantage. section 12.3
CartPole
For CartPole-v1, the actor receives four observations—cart position, cart velocity, pole angle, and pole angular velocity—and returns probabilities for pushing left or right. Training can track episode rewards and plot their trend. section 12.3
class Actor(nn.Module):
def __init__(self, state_dim, action_dim):
super(Actor, self).__init__()
self.fc1 = nn.Linear(state_dim, 64)
self.fc2 = nn.Linear(64, 32)
self.fc3 = nn.Linear(32, action_dim)
def forward(self, state):
x1 = F.relu(self.fc1(state))
x2 = F.relu(self.fc2(x1))
action_probs = F.softmax(self.fc3(x2), dim=-1)
return action_probs
Inspect the compact PyTorch Actor class, including its three-layer architecture, ReLU activations, and softmax output. section 12.3
