liveBook · Manning

AI agent

As explained in Applied Reinforcement Learning

An entity in reinforcement learning that observes an environment, selects actions, and improves its behavior from feedback.

An AI agent makes decisions by interacting with an environment and learning from feedback. For example, a pricing agent can observe inventory and demand, choose a price, and use the resulting profit to improve future decisions. section 1.7 section 2.4

The agent observes a state, selects an action, receives a reward, and encounters a new state. Over repeated interactions, it learns a policy: a mapping from states to actions intended to maximize long-term reward. Actions can affect both the immediate reward and the probability of future states, so a good decision may involve delayed consequences. section 1.7 section 2.3

Figure 1.7 Overview of reinforcement learning framework.

Follow the interaction loop between the agent and its environment. section 1.7

Choosing ads through feedback

A campaign agent chooses among five ads over 1,000 impressions. A click gives a reward of 1; no click gives 0. The simulated environment knows each ad’s true click-through rate, but the agent must estimate those rates from observed outcomes. It must try alternatives without overlooking the ads that already appear promising. section 5.3

Listing 5.4 The Epsilon-Greedy Agent
class EpsilonGreedyAgent:
    def __init__(self, num_ads, epsilon=0.1):
        self.num_ads = num_ads
        self.epsilon = epsilon
        self.arm_counts = [0] * num_ads
        self.q_values = [0.0] * num_ads
 
    def choose_action(self):
        if np.random.rand() < self.epsilon: #A
            return np.random.randint(self.num_ads)
        else:
            return np.argmax(self.q_values) #B
 
    def update(self, chosen_ad_index, reward):
        self.arm_counts[chosen_ad_index] += 1
        n = self.arm_counts[chosen_ad_index]
        current_q = self.q_values[chosen_ad_index]
        new_q = current_q + (1/n) * (reward - current_q)
        self.q_values[chosen_ad_index] = new_q

An epsilon-greedy agent sometimes chooses a random ad and otherwise selects the one with the highest estimated value, updating that estimate after feedback. section 5.3

The agent is only one part of a useful system. Its state must capture information relevant to future decisions: a pricing state that omits inventory or pending backorders can leave the agent acting blindly. Training in a simulated environment also lets an agent make costly or unsafe mistakes without directly exposing a business to them. section 2.4 section 3.0

For applied problems, start with a simple, stable algorithm suited to the action space. Better representations and honest evaluation can matter more than extensive tuning; training should also retain enough exploration to avoid settling prematurely on one action. section 12.9.4

How Applied Reinforcement Learning covers it

How deep it goes
Explains the interaction loop and implements several bandit agents, while emphasizing environment design over extensive agent tuning.
Examples
Uses dynamic pricing, warehouse navigation, and a simulated five-ad campaign to illustrate agent decisions.
What you can do after
Builds a bandit agent from scratch to select ads and update estimates from click feedback.
How it connects
Connects agents to policies, Markov decision processes, state design, rewards, and exploration.

Also explained in

Compare how 46 books explain AI agent →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.