liveBook · Manning

Transformer encoder

As explained in Applied Reinforcement Learning

A neural-network encoder formed from stacked self-attention and feedforward layers with residual connections and normalization.

A transformer encoder lets each zone representation use information from every other zone, supporting context-dependent coordination across a data center. section 9.7

Self-attention uses query, key, and value projections to weight inputs by pairwise relevance. Multiple heads learn different interaction patterns, while stacked layers capture longer-range effects. Positional encoding adds zone identity. section 9.8

Cooling-control network

The cooling agent embeds each zone’s eight-dimensional observation into a 64-dimensional space, then applies two encoder layers with four attention heads and a 128-dimensional feedforward network. section 9.7

Listing 9.6 Transformer actor-critic architecture
class TransformerPolicyValue(nn.Module):
    def __init__(self, config: Config):
        super().__init__()
        self.cfg = config
        obs_dim = 8  #A
        action_dim = 2  #B
 
        self.input_embed = nn.Sequential(
            nn.Linear(obs_dim, config.d_model),
            nn.LayerNorm(config.d_model),
            nn.ReLU())  #C
 
        self.pos_encoding = PositionalEncoding(
            config.d_model, max_len=config.n_zones + 1)  #D
 
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=config.d_model,
            nhead=config.n_heads,
            dim_feedforward=config.d_ff,
            dropout=config.dropout,
            activation='gelu',
            batch_first=True)  #E
        self.transformer = nn.TransformerEncoder(
            encoder_layer, num_layers=config.n_layers)  #F
 
        self.policy_head = nn.Sequential(
            nn.Linear(config.d_model, config.d_model),
            nn.ReLU(),
            nn.Linear(config.d_model, action_dim * 2))  #G
 
        self.value_head = nn.Sequential(
            nn.Linear(config.d_model, config.d_model),
            nn.ReLU(),
            nn.Linear(config.d_model, 1))  #H
 
        self.log_std_min = -5.0  #I
        self.log_std_max = 2.0  #I

Inspect the PyTorch implementation of the encoder within the actor-critic network. section 9.7

How Applied Reinforcement Learning covers it

How deep it goes
Explains attention routing and encoder layer roles, but does not derive the attention calculations.
Examples
Uses two encoder layers with four attention heads to process observations from eight zones.
What you can do after
Apply it via PyTorch's TransformerEncoder to let zone representations attend across a data center.
How it connects
Connects self-attention with feedforward networks, residual connections, layer normalization, and positional encoding.

Also explained in

Compare how 3 books explain transformer encoder →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.