Data exploration means trying new or insufficiently tested options to learn whether they are worth pursuing. In optimization, those options are feasible regions of a search space; in a multi-armed bandit, they are actions whose rewards are not yet well known. The alternative is exploitation: concentrating on an option that already looks promising. section 1.5 section 12.1.6
Imagine moving to a new city. Trying unfamiliar restaurants may uncover a better choice, but it costs visits that could have gone to a known favorite. A search algorithm faces the same trade-off: too little exploration can leave it in a local optimum; too much can waste time when a good solution is already available. section 1.5 section 1.6
Several strategies put that trade-off into practice:
- Explore only: choose an option at random each trial. This tests alternatives but does not increasingly favor a promising one. section 12.1.6
- Epsilon-greedy: explore randomly with probability ε; otherwise choose the option currently treated as best. Adjust ε to change how often the search branches out. section 12.1.6
- Upper confidence bound (UCB): try each option, then select using its average observed reward plus an exploration term. Its parameter c controls that term. section 12.1.6
Three slot machines
Treat each machine as an action and its payout as a reward. A 90-trial NumPy simulation uses three payoff distributions with means of 8, 10, and 5. After each pull, only the selected machine produces a sampled payoff. Compare strategies by average regret—the gap between the best mean payoff and the reward received in this simulation. The supplied greedy and epsilon-greedy listings select using the specified payoff means, so their code illustrates the selection rules rather than learning an unknown best arm from observations. section 12.1.6
def epsilon_greedy(epsilon):
total_regret = 0
for _ in range(num_trials):
if np.random.random() < epsilon:
selected_machine = np.random.randint(K) #1
else:
selected_machine = np.argmax([payoff_params[i]["mean"]
for i in range(K)]) #2
reward = sample_payoff(selected_machine) #3
regret = max_reward - reward
total_regret += regret
average_regret = total_regret / num_trials
return average_regret
#1 Randomly select a slot machine for exploration.
#2 Select the slot machine with the highest mean payoff for exploitation.
#3 Sample the payoff from the selected slot machine.
The epsilon-greedy listing makes the exploration decision visible: a random draw determines whether to select a random machine or the one with the highest specified mean payoff. section 12.1.6
Exploration is not limited to bandits. Adaptive tabu search can promote diversification when improvement stalls or a local optimum appears. In genetic algorithms, mutation introduces variation, while crossover can sometimes create new gene combinations. Which mechanism helps depends on the problem and its settings; experiment with exploration parameters rather than treating a single balance as universally best. section 6.2.3 section 8.9
