liveBook · Manning

Feedforward network

As explained in Applied Reinforcement Learning

A feedforward neural network is a neural network in which information flows from input to output without recurrent connections.

A feedforward network transforms input into output through successive layers, without recurrent connections. In a treatment-planning simulation, it can turn a patient’s current treatment state into scores for the available actions. section 8.4

The treatment state combines features such as cumulative doses, remaining tumor dose, organ safety margins, tumor size, and fraction progress. Normalizing these values keeps the inputs in a reasonable range. The network produces a Q-value for each action; an agent can choose the action with the highest predicted value or explore another action. section 8.4

A feedforward Q-network

A small network takes the state vector through a weighted layer and a ReLU activation, which replaces negative values with zero. A second weighted layer produces the action scores. During training, backpropagation passes the output error backward to update the weights and biases. section 8.4

Listing 8.9 Q-network architecture and forward pass
class MLPQNetwork:
    def __init__(self, input_dim: int, hidden_dim: int,
    ➥output_dim: int, rng=None):
        if rng is None:
            rng = np.random.RandomState(42)
        self.rng = rng
 
        limit1 = np.sqrt(6 / (input_dim + hidden_dim))  #A
        self.W1 = rng.uniform(-limit1, limit1,
        ➥size=(input_dim, hidden_dim)).astype(np.float32)  #B
        self.b1 = np.zeros(hidden_dim, dtype=np.float32)  #C
 
        limit2 = np.sqrt(6 / (hidden_dim + output_dim))  #A
 
        self.W2 = rng.uniform(-limit2, limit2,
        ➥size=(hidden_dim, output_dim)).astype(np.float32)  #B
 
        self.b2 = np.zeros(output_dim, dtype=np.float32)  #C
 
    def forward(self, x):
        z1 = x @ self.W1 + self.b1  #D
        h1 = np.maximum(z1, 0.0)  #E
        q_values = h1 @ self.W2 + self.b2  #F
        cache = (x, z1, h1)  #G
        return q_values, cache  #H

The listing shows the two weighted layers and the ReLU step in the Q-network’s forward pass. section 8.4

Feedforward networks also appear inside transformer blocks. After attention mixes information across token positions, the feedforward component transforms each token representation independently. Residual connections and normalization surround these components. section 10.2

How Applied Reinforcement Learning covers it

How deep it goes
Works through a Q-network’s forward pass and weight updates, while describing the transformer feedforward component only at a high level.
Examples
Uses simulated radiotherapy treatment decisions and contextual token representations in medical questions as examples.
What you can do after
Build it from scratch as a simple Q-network with a forward pass and weight updates.
How it connects
Connects feedforward networks to deep Q-networks, backpropagation, attention, and transformers.

Also explained in

Compare how 20 books explain feedforward network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.