A simulator is a software system that reproduces an environment’s state changes and consequences when an agent acts. In reinforcement learning, the agent practices by taking actions, observing outcomes, and receiving rewards instead of acting in the real world. section 10.5
A useful simulator should let an agent:
- Initialize or reset the environment and agent. section 10.5
- Read the current state after each action. section 10.5
- Apply an action, update the environment, and calculate its reward. section 10.5
- Determine whether the goal is achieved or the simulation should end. section 10.5
Parking-lot navigation
A grid-based car simulator can represent roads, empty spaces, other cars, pedestrians, and the owner’s location. Moving into an empty space earns a reward, while collisions incur penalties. The agent moves north, south, east, or west; the simulator checks boundaries, updates the position, returns a reward, and tests the goal. Poor rewards may produce unwanted behavior, such as moving forward and backward indefinitely. section 10.5
class Simulator:
def __init__(self, road, road_size_x, road_size_y,
➥agent_start_x, agent_start_y):
self.road = road
self.road_size_x = road_size_x
self.road_size_y = road_size_y
self.agent_x = agent_start_x
self.agent_y = agent_start_y #A
def move_agent(self, action):
next_x, next_y = self.agent_x, self.agent_y
if action == COMMAND_NORTH: #B
next_x -= 1
elif action == COMMAND_SOUTH:
next_x += 1
elif action == COMMAND_EAST:
next_y += 1
elif action == COMMAND_WEST:
next_y -= 1
reward_update = 0
if self.is_within_bounds(next_x, next_y): #C
reward_update = self.cost_movement(next_x, next_y) #D
self.agent_x, self.agent_y = next_x, next_y
else:
reward_update = ROAD_OUT_OF_BOUNDS_REWARD #E
return reward_update
Trace a minimal simulator implementation from action commands through bounds checking, movement, and reward updates. section 10.5
