liveBook · Manning

Action space

As explained in Applied Reinforcement Learning

An action space is the set or structured specification of actions that an agent can submit to an environment.

A delivery agent choosing its next customer needs a defined set of possible choices. That set is its action space: the actions the agent can submit to an environment at each decision step. The space should represent real decision options and respect domain constraints. section 3.1 section 3.5

Actions can be discrete, such as choosing a destination, or continuous, such as adjusting a quantity or speed. An action can also combine several fields. The environment uses the submitted action in its transition logic to update the state and determine consequences and rewards. section 3.1 section 3.4

Resource allocation

An action pairs a project ID with an allocation amount; (-1, 0) means stop allocating. The overall list includes positive allocations up to the configured maximum and total budget. At a particular state, an allocation is valid only if it fits the remaining budget and the project has not already been attempted. section 4.4

Listing 4.3 Define Action Space
def create_action_space(mdp: ResourceAllocationMDP) -> list[tuple]:
 
    actions = [(-1, 0)]  #A
    
    for project_id in range(mdp.n_projects): #B
        for allocation in range(1, min(mdp.max_allocation + 1,  #C
  mdp.max_budget + 1)):
            actions.append((project_id, allocation))
    
    return actions

The listing constructs the allocation action list, including the special do-nothing action. section 4.4

Packing a trailer

A placement action is a tuple of container index, x, y, z, and orientation index. A proposed placement must fit within the container and avoid occupied space. A valid placement updates the grid; an invalid one receives a penalty, and repeated failures can cause the item to be skipped. section 3.4

Action-space design also affects which methods fit. A deep Q-network works best with a moderate-sized discrete space because it outputs one value per action; it cannot directly handle continuous actions and struggles with enormous spaces. For portfolio management, choosing among eight named allocation templates instead of raw continuous weights makes decisions readable and supports a categorical-policy PPO. The templates are designed to respect position caps. section 8.5 section 12.4

The decision step need not correspond to a whole business outcome. In language generation, the action space is the vocabulary of possible next tokens, not all possible sentences. A response emerges through repeated token choices. section 10.2

How Applied Reinforcement Learning covers it

How deep it goes
Defines discrete and continuous choices and works through structured action enumeration, state-dependent constraints, and environment validation.
Examples
Uses delivery routing, project allocation, trailer packing, portfolio templates, and next-token selection as action-space examples.
What you can do after
Builds action spaces from scratch with structured choices and constraints suited to a decision environment.
How it connects
Connects action spaces to state spaces, transition dynamics, reward functions, deep Q-networks, and PPO.

Also explained in

Compare how 4 books explain action space →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.