A feedforward network processes each input independently. A recurrent neural network (RNN) instead processes a sequence one step at a time, carrying a state that contains information about earlier inputs. That state lets the current input affect an output in the context of what came before. section 13.3.1
A state passed through a loop
Start with state_t = 0. For each input_t, compute output_t = f(input_t, state_t), then set state_t = output_t. The next step therefore receives both its own input and a value shaped by preceding steps. In the simple NumPy RNN, the state is a vector with one entry per output feature; a tanh calculation combines it with the new input. section 13.3.1
Listing 13.15 NumPy implementation of a simple RNN
import numpy as np timesteps = 100 #1 input_features = 32 #2 output_features = 64 #3 inputs = np.random.random((timesteps, input_features)) #4 state_t = np.zeros((output_features,)) #5 W = np.random.random((output_features, input_features)) #6 U = np.random.random((output_features, output_features)) #6 b = np.random.random((output_features,)) #6 successive_outputs = [] for input_t in inputs: #7 output_t = np.tanh(np.dot(W, input_t) + np.dot(U, state_t) + b) #8 successive_outputs.append(output_t) #9 state_t = output_t #10 final_output_sequence = np.concatenate(successive_outputs, axis=0) #11
The NumPy loop makes the handoff visible: initialize state_t, calculate each output_t using it, then assign that output as the next state. section 13.3.1
State belongs to the sequence being processed. Reset it between independent sequences rather than letting one sample's history leak into another. A recurrent layer can return an output at every timestep, or just a final output containing information about the sequence; return the full sequence when another RNN layer needs to process every step. section 13.3.1 section 20.1.6
Carrying state while generating text
A character-level GRU predicts the next character using information propagated from earlier characters. During training on fixed-length sequences, its state is handled implicitly. To generate one character at a time, use a model that accepts an initial_state and returns an output_state. Feed a prompt through it character by character to establish the starting state; then pass each returned state into the next prediction along with the selected character. section 15.1.1 section 15.1.2
Listing 15.6 Modifying the language model for autoregressive inference
inputs = keras.Input(shape=(1,), dtype="int", name="token_ids") #1 input_state = keras.Input(shape=(hidden_dim,), name="state") x = layers.Embedding(vocabulary_size, embedding_dim)(inputs) x, output_state = layers.GRU(hidden_dim, return_state=True)( x, initial_state=input_state ) outputs = layers.Dense(vocabulary_size, activation="softmax")(x) generation_model = keras.Model( inputs=(inputs, input_state), outputs=(outputs, output_state), ) generation_model.set_weights(model.get_weights()) #2
The generation model exposes both sides of the handoff: a state input and a returned GRU state. section 15.1.2
The starting state need not be zero. In sequence-to-sequence translation, an encoder's representation of the source sentence becomes the decoder's initial state, so decoding begins with source information already available. Also, “state” is broader than recurrent memory: model weights in a stateless JAX training step and accumulators in a Keras metric are called state, but serve different jobs. section 15.2.2 section 3.5.4 section 7.3.1
