A constraint is a rule that limits the decisions an optimization model can make. A business may want to maximize profit, for example, but it must still work within available labor, materials, regulations, and other limits. A useful model balances those realities against the need to remain simple enough to solve. section 1.3 section 1.6.1
A bakery’s limited resources
Suppose chocolate cakes need 2 hours of labor and 3 kg of flour each, while vanilla cakes need 1 hour and 2 kg. With 100 labor hours and 150 kg of flour, the bakery must choose quantities that respect both limits. If x1 and x2 are the numbers of chocolate and vanilla cakes, the constraints are 2x1 + x2 ≤ 100 and 3x1 + 2x2 ≤ 150. Within those limits, the objective is to maximize profit: $5 per chocolate cake and $4 per vanilla cake. section 1.6.1
In reinforcement learning, which actions are valid can depend on the current state. One way to help an agent make context-aware decisions is to include changing constraint information in that state, such as remaining budget, available capacity, or maintenance status. Making information visible is distinct from explicitly restricting the actions the agent may select. section 2.6.2 section 4.4
Listing 2.10 Example of including constraint information in state
Current state = [
server_capacities: [0.7, 0.3, 0.9, ...], #A
memory_available: [16GB, 8GB, 32GB, ...], #B
cooling_zones: [0, 1, 0, 2, ...], #C
power_budget_remaining: 850kW, #D
maintenance_windows: [0, 0, 1, 0, ...], #E
client_priorities: [0.9, 0.5, 0.8, ...], #F
current_temperature: [22, 24, 19, ...] #G
]
The data-center state includes capacity, power, maintenance, and temperature information that can affect allocation decisions. section 2.6.2
Valid actions in resource allocation
An action selects a project and an allocation amount. The amount cannot exceed the remaining budget, must be positive, and can only go to a project that has not already been attempted. A separate action, (-1, 0), means do nothing or cease allocation. section 4.4
Listing 4.7 Policy improvement Function
def policy_improvement(mdp: ResourceAllocationMDP,
states: list[tuple],
value_function: dict[tuple, float]) -> tuple[dict[tuple, tuple], bool]:
actions = create_action_space(mdp)
new_policy = {}
policy_stable = True #A
for state in states:
budget, proj_outcomes = state
valid_actions = [] #B
for action in actions: #B
project_id, allocation = action #B
if project_id == -1: #B
valid_actions.append(action) #B
elif (allocation <= budget and #B
project_id < mdp.n_projects and #B
proj_outcomes[project_id] == 0): #B
valid_actions.append(action) #B
best_action = None #C
best_value = float('-inf') #C
for action in valid_actions:
expected_value = 0.0
for next_state in states: #D
trans_prob = transition_probability(mdp, state, action, next_state) #D
if trans_prob > 0:
reward = reward_function(mdp, state, action, next_state) #D
expected_value += trans_prob * (reward + mdp.discount_factor * value_function[next_state]) #D
if expected_value > best_value: #E
best_value = expected_value #E
best_action = action #E
new_policy[state] = best_action if best_action else (-1, 0) #F
return new_policy, policy_stable
The policy improvement function filters actions against the current budget and project outcomes before choosing among valid actions. section 4.4




























