liveBook · Manning

Environment

As explained in Applied Reinforcement Learning

An environment is the external system or setting with which an agent interacts and whose responses produce consequences for the agent's actions.

A pricing agent does not make decisions in isolation: customers respond, demand changes, and each price has consequences. The environment is the external system or setting the agent interacts with. Through repeated actions and feedback, the agent can learn a policy for choosing what to do. section 1.1 section 1.7

At each decision, the agent acts in a state and observes a reward and a subsequent state. In a Markov decision process, the action influences both the immediate reward and the probability of moving to the next state. An environment therefore makes the consequences of decisions available for learning, rather than merely supplying data for prediction. section 2.3

Direct trial and error can be costly or unsafe in a business. A simulated environment provides a sandbox that mirrors essential dynamics, constraints, and feedback, allowing an agent to make mistakes before real-world use. Its usefulness depends on the choices made when modeling the problem; an algorithm alone does not choose the right state or reward. section 3.0 section 12.2

To design a business environment, turn a broad goal into a manageable decision process:

  • Scope a bounded problem. For last-mile delivery, start with a driver's route in a given time window rather than optimizing every logistics decision at once. section 3.1
  • Identify entities and assumptions, such as vehicles, customers, depots, traffic zones, and delivery time windows. section 3.1
  • Define the state, actions, transitions, rewards, and constraints that make the process simulatable. Tie rewards to the business outcome being sought. section 2.3 section 3.0 section 3.2
Figure 3.1 Conceptual framework for designing business optimization environments

The framework connects business problem framing to the design of an optimization environment. section 3.1

Warehouse order picking

A robot collects items on a 2D grid and returns to a depot. It moves one cell at a time; item and obstacle locations are fixed for an order. Its state tracks position, items collected, and whether it is time to return. Fewer steps indicate greater efficiency, so rewards can encourage short paths while penalizing collisions and incomplete orders. Resetting restores the starting configuration for a new episode. section 3.2

Table 3.1 Key reward signals used in warehouse order picking environment
Agent behavior
Reward
Purpose
Move to a valid (empty) cell
–0.1
Small step penalty to encourage shorter, more efficient paths
Hit wall (outside grid)
–0.5
Penalize invalid moves outside the grid limits
Attempt to move into a blocked cell (hit obstacle)
–0.7
Stronger penalty to discourage unsafe or infeasible paths
Successfully pick up an item
+15.0
Significant reward for achieving one of the primary objectives
Return to the depot after collecting all items
+100.0
Large terminal reward for completing the full task successfully
Episode ends without full completion (e.g., time limit reached)
–20.0
Penalty for failing to complete the task within the allowed steps

The reward table shows how movement, collisions, item pickup, and task completion translate operational priorities into feedback. section 3.2

Ad campaign simulation

A simpler environment compares ad-selection strategies. Each ad has a click probability; selecting it produces a simulated reward of 1 for a click or 0 otherwise. Running different agents against the same environment makes their exploration and exploitation behavior comparable. section 5.3

A simulation is only one part of a working system. State representation, reward design, data, and evaluation all need attention: for example, using tomorrow's price in a portfolio feature can make a backtest impossible to reproduce live. section 12.2

How Applied Reinforcement Learning covers it

How deep it goes
Develops a design framework and a warehouse implementation, including state, reset, movement, rewards, and a random-agent sanity check.
Examples
Uses warehouse order picking and an ad campaign simulation to illustrate sequential and one-step environments.
What you can do after
Build environments from scratch by scoping a business process and implementing its state, transitions, and feedback.
How it connects
Connects environments to Markov decision processes, states, actions, rewards, policies, and exploration–exploitation.

Also explained in

Compare how 10 books explain environment →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.