liveBook · Manning

Constraint

As explained in Applied Reinforcement Learning

A restriction that limits the values or combinations allowed in an optimization problem, including which actions are valid in a state.

A constraint is a rule that limits the decisions an optimization model can make. A business may want to maximize profit, for example, but it must still work within available labor, materials, regulations, and other limits. A useful model balances those realities against the need to remain simple enough to solve. section 1.3 section 1.6.1

A bakery’s limited resources

Suppose chocolate cakes need 2 hours of labor and 3 kg of flour each, while vanilla cakes need 1 hour and 2 kg. With 100 labor hours and 150 kg of flour, the bakery must choose quantities that respect both limits. If x1 and x2 are the numbers of chocolate and vanilla cakes, the constraints are 2x1 + x2 ≤ 100 and 3x1 + 2x2 ≤ 150. Within those limits, the objective is to maximize profit: $5 per chocolate cake and $4 per vanilla cake. section 1.6.1

In reinforcement learning, which actions are valid can depend on the current state. One way to help an agent make context-aware decisions is to include changing constraint information in that state, such as remaining budget, available capacity, or maintenance status. Making information visible is distinct from explicitly restricting the actions the agent may select. section 2.6.2 section 4.4

Listing 2.10 Example of including constraint information in state
Current state = [
    server_capacities: [0.7, 0.3, 0.9, ...],      #A
 
    memory_available: [16GB, 8GB, 32GB, ...],     #B
 
    cooling_zones: [0, 1, 0, 2, ...],             #C
 
    power_budget_remaining: 850kW,                #D
 
    maintenance_windows: [0, 0, 1, 0, ...],       #E
 
    client_priorities: [0.9, 0.5, 0.8, ...],      #F
 
    current_temperature: [22, 24, 19, ...]  #G
]

The data-center state includes capacity, power, maintenance, and temperature information that can affect allocation decisions. section 2.6.2

Valid actions in resource allocation

An action selects a project and an allocation amount. The amount cannot exceed the remaining budget, must be positive, and can only go to a project that has not already been attempted. A separate action, (-1, 0), means do nothing or cease allocation. section 4.4

Listing 4.7 Policy improvement Function
def policy_improvement(mdp: ResourceAllocationMDP, 
                      states: list[tuple], 
                      value_function: dict[tuple, float]) -> tuple[dict[tuple, tuple], bool]:
    
    actions = create_action_space(mdp)
    new_policy = {}
    policy_stable = True #A
    
    for state in states:
        budget, proj_outcomes = state
 
        valid_actions = [] #B
        for action in actions: #B
            project_id, allocation = action #B
 
            if project_id == -1: #B
                valid_actions.append(action) #B
            elif (allocation <= budget and  #B
                  project_id < mdp.n_projects and  #B
                  proj_outcomes[project_id] == 0): #B
                valid_actions.append(action) #B
 
        best_action = None #C 
        best_value = float('-inf') #C
        
        for action in valid_actions:
            expected_value = 0.0
            for next_state in states: #D
                trans_prob = transition_probability(mdp, state, action, next_state) #D
                if trans_prob > 0:
                    reward = reward_function(mdp, state, action, next_state) #D
                    expected_value += trans_prob * (reward + mdp.discount_factor * value_function[next_state]) #D
            
            if expected_value > best_value: #E
                best_value = expected_value #E
                best_action = action #E
        
        new_policy[state] = best_action if best_action else (-1, 0) #F
    
    return new_policy, policy_stable

The policy improvement function filters actions against the current budget and project outcomes before choosing among valid actions. section 4.4

How Applied Reinforcement Learning covers it

How deep it goes
Works from a linear resource formulation to state encoding and code that filters valid actions.
Examples
Uses bakery labor and flour limits, data-center maintenance and power budgets, and project allocation rules.
What you can do after
Builds constraint-aware action selection from scratch for a resource allocation model.
How it connects
Connects constraints to decision variables, objective functions, state representation, and policy improvement.

Also explained in

Compare how 29 books explain constraint →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.