concept encoder in category deep learning

This is an excerpt from Manning's book Deep Learning Design Patterns MEAP V02. Login to get full access to this book.

The encoder performs representational learning on the input to learn a function f(x) = x’.

Below is an example implementation of the encoder for an autoencoder to learn the identity function, where we progressively pool the number of nodes (hidden units) through the parameter layers, as depicted in figure 6.1. The output from the encoder is the latent space. We start by flattening the image input to a 1D vector. The parameter layers is a list, where the number of elements is the number of hidden layers and the element value is the number of units at that layer. Since we are progressively pooling, each subsequent element value is progressively smaller. While an encoder tends to be shallow in layers when compared to a CNN used for classification, we add batch normalization for it’s regularizing effect.

def encoder(x, layers):
    ''' Construct the Encoder
        x     : input to the encoder
        layers: number of nodes per layer
    '''
    x = Flatten()(x)    #A
 
    for layer in layers:    #B
        n_nodes = layer['n_nodes']
        x = Dense(n_nodes)(x)
        x = BatchNormalization()(x)
        x = ReLU()(x)
 
    return x    #C

copy

Fig. 6.2 An AutoEncoder learns two functions: the encoder learns to convert a high dimensional representation to a low dimensional representation, and then a decoder learns to reconstruct back to a high dimensional representation that is a translation of the input.

This is an excerpt from Manning's book Deep Reinforcement Learning in Action. Login to get full access to this book.

Figure 8.10 shows the overall graph structure: the forward pass of the components and also the backward (backpropagation) pass to update the model parameters. It is worth repeating that the inverse model backpropagates back through to the encoder model, and the encoder model is only trained together with the inverse model. We must use PyTorch’s detach() method to detach the forward model from the encoder so it won’t backpropagate into the encoder. The purpose of the encoder is not to give us a low-dimensional input for improved performance but to learn to encode the state using a representation that only contains information relevant for predicting actions. This means that aspects of the state that are randomly fluctuating and have no impact on the agent’s actions will be stripped from this encoded representation. This, in theory, should avoid the noisy TV problem.

Figure 8.10. The curiosity module. First the encoder will encode states S_t and S_t+1 into low-dimensional vectors, φ(S_t) and φ(S_t+1) respectively. These encoded states are passed to the forward and inverse models. Notice that the inverse model backpropagates to the encoded model, thereby training it through its own error. The forward model is trained by backpropagating from its own error function, but it does not backpropagate through to the encoder like the inverse model does. This ensures that the encoder learns to produce state representations that are only useful for predicting which action was taken. The black circle indicates a copy operation that copies the output from the encoder and passes the copies to the forward and inverse models.

The encoder is a neural network composed of four convolutional layers (with an identical architecture to the DQN), with an ELU activation function after each layer. The final output is then flattened to get a flat 288-dimensional vector output.

The intrinsic curiosity module (ICM) consists of three independent neural networks: a forward-prediction model, an inverse model, and an encoder.

The encoder encodes high-dimensional states into a low-dimensional vector with high-level features (which removes noise and trivial features).