liveBook · Manning

Transformer architecture

As explained in Applied Reinforcement Learning

A neural-network architecture built around self-attention, position-wise feedforward transformations, residual connections, and normalization.

A transformer is a neural-network architecture built around multi-head attention. It is useful when a decision depends on relationships among inputs: a cooling controller, for example, must account for heat moving between data center zones rather than treating each zone independently. section 9.4 section 9.6

A transformer encoder layer combines two sub-layers: multi-head self-attention enriches each zone’s representation using information from other zones, and a position-wise feedforward network processes each enriched representation independently. section 9.6

Each sub-layer uses a residual connection and layer normalization: Output = LayerNorm(x + SubLayer(x)). Adding the input back preserves existing information and gives gradients a direct path through the network. Normalization re-centers and rescales activations to help keep their scale stable. section 9.6

Stacking encoder layers produces deeper representations. In cooling, one layer might capture direct relationships between zones; a later layer can use those representations to capture indirect relationships. section 9.6

Figure 9.8 Transformer architecture.

Follow the encoder architecture and its sub-layers in the diagram. section 9.6

Coordinating data center cooling

A REINFORCE agent controls eight interconnected zones arranged in a ring. Each zone supplies eight features, including temperature, workload, neighbor temperature, and previous actions. The agent chooses a fan speed and chilled-water valve opening for each zone, balancing energy use against thermal safety. A transformer encoder lets the agent represent relationships among zones when coordinating those actions. section 9.7 section 9.4

Listing 9.6 Transformer actor-critic architecture
class TransformerPolicyValue(nn.Module):
    def __init__(self, config: Config):
        super().__init__()
        self.cfg = config
        obs_dim = 8  #A
        action_dim = 2  #B
 
        self.input_embed = nn.Sequential(
            nn.Linear(obs_dim, config.d_model),
            nn.LayerNorm(config.d_model),
            nn.ReLU())  #C
 
        self.pos_encoding = PositionalEncoding(
            config.d_model, max_len=config.n_zones + 1)  #D
 
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=config.d_model,
            nhead=config.n_heads,
            dim_feedforward=config.d_ff,
            dropout=config.dropout,
            activation='gelu',
            batch_first=True)  #E
        self.transformer = nn.TransformerEncoder(
            encoder_layer, num_layers=config.n_layers)  #F
 
        self.policy_head = nn.Sequential(
            nn.Linear(config.d_model, config.d_model),
            nn.ReLU(),
            nn.Linear(config.d_model, action_dim * 2))  #G
 
        self.value_head = nn.Sequential(
            nn.Linear(config.d_model, config.d_model),
            nn.ReLU(),
            nn.Linear(config.d_model, 1))  #H
 
        self.log_std_min = -5.0  #I
        self.log_std_max = 2.0  #I

Inspect the PyTorch actor-critic setup: per-zone input embedding, positional encoding, and stacked transformer encoder layers. section 9.7

Transformers also appear in language models. There, input text is split into tokens and mapped to numerical IDs before flowing through the model to help predict the next token. section 10.2

How Applied Reinforcement Learning covers it

How deep it goes
Explains encoder sub-layers, residual connections, normalization, and stacking, and provides a PyTorch actor-critic implementation.
Examples
Illustrates an eight-zone data center cooling controller and next-token prediction from tokenized text.
What it assumes
Assumes familiarity with neural networks, attention, policy gradients, and sequence models for the language-model overview.
What you can do after
Apply it via PyTorch encoder layers in a transformer-based actor-critic for coordinated cooling.
How it connects
Connects self-attention and positional encoding with REINFORCE, actor-critic methods, and language models.

Also explained in

Compare how 40 books explain transformer architecture →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.