liveBook · Manning

Discrete action bottleneck

As explained in Applied Reinforcement Learning

The discrete action bottleneck is the scalability limitation caused by representing continuous or high-dimensional controls as a finite set of discrete actions.

The discrete action bottleneck occurs when continuous or high-dimensional controls are represented as a finite set of choices. As controls multiply, even a few levels per control can create an action space too large for value-based methods to handle. section 9.1

Air-conditioning controls

With two controls in each of eight zones, five levels per control create about 152 billion actions. Coarsening to three levels still produces about 43 million choices and prevents nuanced settings, such as a fan speed of 0.73. section 9.1

How Applied Reinforcement Learning covers it

How deep it goes
Quantifies action-space growth and illustrates lost control nuance, without detailing an implementation.
Examples
Uses multi-zone air-conditioning controls and fan-speed settings to demonstrate discretization costs.
What you can do after
Understand why discretizing many continuous controls can make value-based action selection impractical.
How it connects
Connects the bottleneck to value-based methods, Q-networks, continuous control, and policy gradients.

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.