liveBook · Manning

Tabular method

As explained in Applied Reinforcement Learning

Tabular reinforcement learning stores value estimates or policy information explicitly in lookup tables indexed by states or state-action pairs.

A restaurant deciding whether to seat a party now or wait for a larger one can keep a separate estimate for each combination of time remaining and action. A tabular reinforcement learning method stores these estimates in a lookup table: typically, each row represents a state, each column an action, and each cell an action value Q(s, a). section 6.4

Table 6.2. Sample Q-table for Restaurant Scheduling Problem
Hours Remaining (State)
Seat Immediately (Action 0)
Wait for Large Party (Action 1)
0 (Closed)
0.00
0.00
1 hour left
50.00
36.00
2 hours left
50.00
67.50
3 hours left
50.00
78.53
4 hours left
50.00
85.47

Compare the two action values in each row. With one hour left, seating immediately has the higher estimate; with two hours left, waiting does. section 6.4

The table is a representation, not a single learning algorithm. Temporal-difference learning updates estimates using rewards and estimates of future value. Q-learning uses the best next action value for its update, even when it explores another action. SARSA uses the value of the next action actually taken, including exploratory actions. section 6.4 section 6.7

Fuel purchase scheduling

A gas station must order fuel while balancing holding costs against stockout penalties under uncertain demand and delivery delays. A compact table key combines current inventory, the total quantity on order, and the time until the next arrival. An epsilon-greedy policy can explore different orders while usually choosing the action with the highest current estimate. section 6.6

Listing 6.5 State representation with feature engineering\
@dataclass
class EnvironmentState:
    """State representation for the gas station environment."""
    current_inventory: int                #A
    pending_orders: List[Tuple[int, int]] #B
    
    def to_tuple(self) -> Tuple:
        """Convert state to tuple for Q-table indexing."""
        inventory = self.current_inventory #C
        
        pending_sum = sum(quantity for quantity, days_left 
                         in self.pending_orders) #D
        
        if self.pending_orders:
            next_arrival_time = min(days_left for _, days_left 
                                   in self.pending_orders) #E
        else:
            next_arrival_time = 0 #F
        
        return (inventory, pending_sum, next_arrival_time) #G

The state representation converts inventory and pending deliveries into a tuple used for Q-table indexing. section 6.6

Tabular methods fit problems where states and actions can be enumerated and visited repeatedly. Adding state features expands the table and increases training demands. Continuous, high-dimensional, or combinatorial states can make explicit storage impractical; function approximation instead learns parameters that generalize value estimates across states. section 6.6 section 8.1

How Applied Reinforcement Learning covers it

How deep it goes
Defines Q-tables, contrasts update logic, and implements agents in a simulated inventory problem before introducing function approximation.
Examples
Uses restaurant seating decisions and a gas station fuel-ordering simulation with delayed deliveries.
What it assumes
Assumes familiarity with on-policy and off-policy learning before defining tabular representation.
What you can do after
Build tabular agents from scratch for a simulated inventory problem and evaluate their rewards, profits, and stockouts.
How it connects
Connects temporal-difference learning, Q-learning, SARSA, epsilon-greedy exploration, eligibility traces, and function approximation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.