liveBook · Manning

Resource allocation problem

As explained in Applied Reinforcement Learning

A resource allocation problem is a decision problem that assigns limited resources among competing activities to optimize an objective subject to constraints.

A resource allocation problem assigns a limited budget across competing projects whose costs, returns, and chances of success differ. Because outcomes are uncertain and spending affects what remains, the decision can be modeled as a Markov decision process (MDP). section 4.4

Represent each state with the remaining budget and a status for each project. Actions either stop or allocate funds to an eligible project; transitions capture possible outcomes, and rewards balance returns against allocation costs and losses. Dynamic programming evaluates and improves a policy to find allocations with higher expected value. section 4.4

Startup project portfolio

A startup chooses among a predictable product improvement, a high-risk innovation, a very safe maintenance project, and a moderate-risk market expansion. Each competes for a limited budget, and results arrive only after funds are committed. section 4.4

Listing 4.1 Defining Problem Class
class ResourceAllocationMDP:
   
    n_projects: int = 4          #A
    max_budget: int = 10         #B
    max_allocation: int = 5      #C
    discount_factor: float = 0.9 #D
    
    def __post_init__(self):
       
        self.project_names = [f"Project_{chr(65+i)}" for i in range(self.n_projects)] #E
        
       
        self.project_params = { #F
            0: {"base_return": 2.1, "volatility": 0.3, "success_prob": 0.7},
            1: {"base_return": 2.9, "volatility": 0.5, "success_prob": 0.5},
            2: {"base_return": 1.8, "volatility": 0.2, "success_prob": 0.8},
            3: {"base_return": 2.3, "volatility": 0.4, "success_prob": 0.6},
        }

Inspect the Python class defining the number of projects, budget limits, allocation limits, discount factor, and project parameters. section 4.4

How Applied Reinforcement Learning covers it

How deep it goes
Defines states, actions, transitions, and rewards, then implements policy evaluation, improvement, and generalized policy iteration.
Examples
Uses a startup choosing among four projects with distinct risk and return profiles.
What you can do after
Build it from scratch as a finite MDP and compute an allocation policy with generalized policy iteration.
How it connects
Connects resource allocation to Markov decision processes, dynamic programming, policy evaluation, and policy improvement.

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.