A restaurant deciding whether to seat a party now or wait for a larger one can keep a separate estimate for each combination of time remaining and action. A tabular reinforcement learning method stores these estimates in a lookup table: typically, each row represents a state, each column an action, and each cell an action value Q(s, a). section 6.4
Table 6.2. Sample Q-table for Restaurant Scheduling Problem
|
Hours Remaining (State)
|
Seat Immediately (Action 0)
|
Wait for Large Party (Action 1)
|
|
0 (Closed)
|
0.00
|
0.00
|
|
1 hour left
|
50.00
|
36.00
|
|
2 hours left
|
50.00
|
67.50
|
|
3 hours left
|
50.00
|
78.53
|
|
4 hours left
|
50.00
|
85.47
|
Compare the two action values in each row. With one hour left, seating immediately has the higher estimate; with two hours left, waiting does. section 6.4
The table is a representation, not a single learning algorithm. Temporal-difference learning updates estimates using rewards and estimates of future value. Q-learning uses the best next action value for its update, even when it explores another action. SARSA uses the value of the next action actually taken, including exploratory actions. section 6.4 section 6.7
Fuel purchase scheduling
A gas station must order fuel while balancing holding costs against stockout penalties under uncertain demand and delivery delays. A compact table key combines current inventory, the total quantity on order, and the time until the next arrival. An epsilon-greedy policy can explore different orders while usually choosing the action with the highest current estimate. section 6.6
Listing 6.5 State representation with feature engineering\
@dataclass
class EnvironmentState:
"""State representation for the gas station environment."""
current_inventory: int #A
pending_orders: List[Tuple[int, int]] #B
def to_tuple(self) -> Tuple:
"""Convert state to tuple for Q-table indexing."""
inventory = self.current_inventory #C
pending_sum = sum(quantity for quantity, days_left
in self.pending_orders) #D
if self.pending_orders:
next_arrival_time = min(days_left for _, days_left
in self.pending_orders) #E
else:
next_arrival_time = 0 #F
return (inventory, pending_sum, next_arrival_time) #G
The state representation converts inventory and pending deliveries into a tuple used for Q-table indexing. section 6.6
Tabular methods fit problems where states and actions can be enumerated and visited repeatedly. Adding state features expands the table and increases training demands. Continuous, high-dimensional, or combinatorial states can make explicit storage impractical; function approximation instead learns parameters that generalize value estimates across states. section 6.6 section 8.1
