An action is a choice an agent makes while interacting with an environment. In a Markov decision process (MDP), the agent selects an action from the permissible action set, receives a reward, and moves to a new state. A policy maps states—or the observations available to the agent—to actions. Learning aims to choose actions that maximize cumulative reward, not just the next reward. section 12.1.2 section 12.1
Choose a commute
From home, choosing the train, car, or bike is an action. If travel were deterministic, each choice would have a predictable result. Under uncertainty, taking the train might get you to work on time or leave you waiting for the next one; driving can produce different travel times as traffic changes. The choice therefore needs to account for possible outcomes, not a single promised arrival time. section 12.1.1
What counts as an action depends on the problem:
- In an MDP, actions affect the environment and may have consequences across multiple time steps. Actions can be discrete, as in a board game, or continuous, as in robot control. section 12.1 section 12.1.2
- In a multi-armed bandit, selecting an arm produces a reward but does not change the environment’s state. The task is to balance trying arms against choosing a promising one. section 12.1.6
- In a contextual bandit, the best arm can change with the current context; expected reward depends on both the context and the chosen action. section 12.1
def epsilon_greedy(epsilon):
total_regret = 0
for _ in range(num_trials):
if np.random.random() < epsilon:
selected_machine = np.random.randint(K) #1
else:
selected_machine = np.argmax([payoff_params[i]["mean"]
for i in range(K)]) #2
reward = sample_payoff(selected_machine) #3
regret = max_reward - reward
total_regret += regret
average_regret = total_regret / num_trials
return average_regret
#1 Randomly select a slot machine for exploration.
#2 Select the slot machine with the highest mean payoff for exploitation.
#3 Sample the payoff from the selected slot machine.
The epsilon-greedy listing makes action selection concrete: sometimes it selects a slot machine randomly, and otherwise it selects the one with the highest specified mean payoff. Compare those two branches to see the exploration–exploitation choice; the listing uses supplied payoff means for its exploitation branch. section 12.1.6
For a sequential optimization problem, defining the actions also defines the decisions an agent can make while constructing a solution. Routing problems are examples of this RL framing: the goal is to learn a policy that maps states to choices yielding strong long-term rewards. Start by identifying the permissible choices, then ask whether each choice changes a future state; that distinction helps determine whether an MDP or bandit model fits. section 12.2 section 12.1
