A transformer encoder lets each zone representation use information from every other zone, supporting context-dependent coordination across a data center. section 9.7
Self-attention uses query, key, and value projections to weight inputs by pairwise relevance. Multiple heads learn different interaction patterns, while stacked layers capture longer-range effects. Positional encoding adds zone identity. section 9.8
Cooling-control network
The cooling agent embeds each zone’s eight-dimensional observation into a 64-dimensional space, then applies two encoder layers with four attention heads and a 128-dimensional feedforward network. section 9.7
Listing 9.6 Transformer actor-critic architecture
class TransformerPolicyValue(nn.Module):
def __init__(self, config: Config):
super().__init__()
self.cfg = config
obs_dim = 8 #A
action_dim = 2 #B
self.input_embed = nn.Sequential(
nn.Linear(obs_dim, config.d_model),
nn.LayerNorm(config.d_model),
nn.ReLU()) #C
self.pos_encoding = PositionalEncoding(
config.d_model, max_len=config.n_zones + 1) #D
encoder_layer = nn.TransformerEncoderLayer(
d_model=config.d_model,
nhead=config.n_heads,
dim_feedforward=config.d_ff,
dropout=config.dropout,
activation='gelu',
batch_first=True) #E
self.transformer = nn.TransformerEncoder(
encoder_layer, num_layers=config.n_layers) #F
self.policy_head = nn.Sequential(
nn.Linear(config.d_model, config.d_model),
nn.ReLU(),
nn.Linear(config.d_model, action_dim * 2)) #G
self.value_head = nn.Sequential(
nn.Linear(config.d_model, config.d_model),
nn.ReLU(),
nn.Linear(config.d_model, 1)) #H
self.log_std_min = -5.0 #I
self.log_std_max = 2.0 #I
Inspect the PyTorch implementation of the encoder within the actor-critic network. section 9.7


