A pricing agent does not make decisions in isolation: customers respond, demand changes, and each price has consequences. The environment is the external system or setting the agent interacts with. Through repeated actions and feedback, the agent can learn a policy for choosing what to do. section 1.1 section 1.7
At each decision, the agent acts in a state and observes a reward and a subsequent state. In a Markov decision process, the action influences both the immediate reward and the probability of moving to the next state. An environment therefore makes the consequences of decisions available for learning, rather than merely supplying data for prediction. section 2.3
Direct trial and error can be costly or unsafe in a business. A simulated environment provides a sandbox that mirrors essential dynamics, constraints, and feedback, allowing an agent to make mistakes before real-world use. Its usefulness depends on the choices made when modeling the problem; an algorithm alone does not choose the right state or reward. section 3.0 section 12.2
To design a business environment, turn a broad goal into a manageable decision process:
- Scope a bounded problem. For last-mile delivery, start with a driver's route in a given time window rather than optimizing every logistics decision at once. section 3.1
- Identify entities and assumptions, such as vehicles, customers, depots, traffic zones, and delivery time windows. section 3.1
- Define the state, actions, transitions, rewards, and constraints that make the process simulatable. Tie rewards to the business outcome being sought. section 2.3 section 3.0 section 3.2
Figure 3.1 Conceptual framework for designing business optimization environments
The framework connects business problem framing to the design of an optimization environment. section 3.1
Warehouse order picking
A robot collects items on a 2D grid and returns to a depot. It moves one cell at a time; item and obstacle locations are fixed for an order. Its state tracks position, items collected, and whether it is time to return. Fewer steps indicate greater efficiency, so rewards can encourage short paths while penalizing collisions and incomplete orders. Resetting restores the starting configuration for a new episode. section 3.2
Table 3.1 Key reward signals used in warehouse order picking environment
|
Agent behavior
|
Reward
|
Purpose
|
|
Move to a valid (empty) cell
|
–0.1
|
Small step penalty to encourage shorter, more efficient paths
|
|
Hit wall (outside grid)
|
–0.5
|
Penalize invalid moves outside the grid limits
|
|
Attempt to move into a blocked cell (hit obstacle)
|
–0.7
|
Stronger penalty to discourage unsafe or infeasible paths
|
|
Successfully pick up an item
|
+15.0
|
Significant reward for achieving one of the primary objectives
|
|
Return to the depot
after collecting all items
|
+100.0
|
Large terminal reward for completing the full task successfully
|
|
Episode ends without full completion (e.g., time limit reached)
|
–20.0
|
Penalty for failing to complete the task within the allowed steps
|
The reward table shows how movement, collisions, item pickup, and task completion translate operational priorities into feedback. section 3.2
Ad campaign simulation
A simpler environment compares ad-selection strategies. Each ad has a click probability; selecting it produces a simulated reward of 1 for a click or 0 otherwise. Running different agents against the same environment makes their exploration and exploitation behavior comparable. section 5.3
A simulation is only one part of a working system. State representation, reward design, data, and evaluation all need attention: for example, using tomorrow's price in a portfolio feature can make a backtest impossible to reproduce live. section 12.2









