liveBook · Manning

Step function

As explained in Applied Reinforcement Learning

A step function applies an agent action to an environment and returns the resulting observation, feedback, and episode status.

A step function applies an agent’s action to the current environment state, computes a reward, updates the state, and returns the next state, reward, episode status, and an empty information dictionary. The template refuses calls after the environment is done; its example marks the episode done after each step. section 3.1

Listing 3.1 Template for any reinforcement learning environment
class CustomEnv:
    def __init__(self): #A
 
        self.current_step = 0 #A
        self.done = False #A
        self.state = None #A
        self.action_space = [0,1] #B
        self.observation_space = np.zeros(3, dtype=np.float32) #B
 
    def reset(self): #C
 
        self.current_step = 0 #C
        self.done = False #C
        self.state = self._get_initial_state() #C
        return self.state #C
 
    def step(self, action):
 
        if self.done: #D
            raise Exception("Environment is done. Call reset().") #D
        next_state = self._transition(self.state, action) #E
        reward = self._compute_reward(self.state, action, next_state) #F
        self.state = next_state #G
        self.current_step += 1 #G
        self.done = True #G
        return next_state, reward, self.done, {} #H
 
    def _get_initial_state(self): #I
        raise NotImplementedError #I
 
    def _transition(self, state, action): #J
        raise NotImplementedError #J
 
    def _compute_reward(self, state, action, next_state): #K
        raise NotImplementedError #K

See the environment template for the step method’s transition, reward, state-update, and return sequence. section 3.1

How Applied Reinforcement Learning covers it

How deep it goes
Shows a step implementation that delegates transitions and rewards to helpers, without detailing those calculations.
Examples
Uses a Python environment template with binary actions, a three-value observation space, and a step method that ends the episode.
What you can do after
Build a custom environment step method that applies an action, calculates feedback, updates state, and returns episode status.
How it connects
Connects reinforcement learning environments with states, actions, transitions, and rewards.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.