concept Autoencoder in category machine learning

appears as: Autoencoder
Machine Learning with TensorFlow, Second Edition

This is an excerpt from Manning's book Machine Learning with TensorFlow, Second Edition.

Finally, create a new Python source file called main.py, and use your Autoencoder class, as shown in the following listing.

The CIFAR-10 Autoencoder class that you built last chapter had a single hidden layer of size 100 neurons, and took as input a 5000 x 1024 vector of training images, and encoded the input using tf.matmul and a hidden layer of size 1024 x 100, and decoded the hidden layer using an output layer of 100 x 1024. The Autoencoder class trained using 1000 epochs and a user-specified batch size and also provided methods to test the process by printing the values of the hidden neurons – or the encoding – and then the decoded representation of the original input values. Listing 12.1 shows the class and its utility methods in one easy reference.

Listing 12.1 The Autoencoder class
def get_batch(X, size):
    a = np.random.choice(len(X), size, replace=False)#A
    return X[a]#A
 
class Autoencoder:
    def __init__(self, input_dim, hidden_dim, epoch=1000, batch_size=50, learning_rate=0.001):
        self.epoch = epoch
        self.batch_size = batch_size
        self.learning_rate = learning_rate
 
        x = tf.placeholder(dtype=tf.float32, shape=[None, input_dim])
        with tf.name_scope('encode'):#B
            weights = tf.Variable(tf.random_normal([input_dim, hidden_dim], dtype=tf.float32), name='weights')
            biases = tf.Variable(tf.zeros([hidden_dim]), name='biases')
            encoded = tf.nn.sigmoid(tf.matmul(x, weights) + biases)#C
        with tf.name_scope('decode'):
            weights = tf.Variable(tf.random_normal([hidden_dim, input_dim], dtype=tf.float32), name='weights')
            biases = tf.Variable(tf.zeros([input_dim]), name='biases')
            decoded = tf.matmul(encoded, weights) + biases#D
 
        self.x = x
        self.encoded = encoded
        self.decoded = decoded
 
        self.loss = tf.sqrt(tf.reduce_mean(tf.square(tf.subtract(self.x, self.decoded))))#E
        self.train_op = tf.train.RMSPropOptimizer(self.learning_rate).minimize(self.loss)#E
        
        self.saver = tf.train.Saver()#F
 
    def train(self, data):
        with tf.Session() as sess:
            sess.run(tf.global_variables_initializer())
            for i in range(self.epoch):
                for j in range(np.shape(data)[0] // self.batch_size):#G
                    batch_data = get_batch(data, self.batch_size)
                    l, _ = sess.run([self.loss, self.train_op], feed_dict={self.x: batch_data})
                if i % 10 == 0:
                    print('epoch {0}: loss = {1}'.format(i, l))
                    self.saver.save(sess, './model.ckpt')
            self.saver.save(sess, './model.ckpt')
        
    def test(self, data):
        with tf.Session() as sess:
            self.saver.restore(sess, './model.ckpt')
            hidden, reconstructed = sess.run([self.encoded, self.decoded], feed_dict={self.x: data})#H
        print('input', data)#H
        print('compressed', hidden)#H
        print('reconstructed', reconstructed)#H
        return reconstructed

Though you ran this class in the last chapter, you didn’t take a look at the input to it or the output from the decoding step to check out how well it learned a representation of the input. To do this, you can use the code leveraged in the last chapter, which is recommended on the CIFAR-10 website as an easy way to read the Python-formatted data stored in the Pickle format. Pickle is a Python library for compact binary representation by serializing a Python object into a byte stream and then providing methods to deserialize the object back into an active Python dynamic object from that byte stream. You can use the unpickle function and associated dataset loading code like the greyscale function that coverts the 50000 training 3-channel RGB images into 1 channel grayscale images by taking the mean of the RGB values for each image. The rest of the CIFAR-10 loading code iterates through the downloaded five 10,000-image training files. The files are in the Python pickle format where the data is stored in a Python dictionary with key data and the key labels with valid values 0-9 for each of the ten image classes. The classes index into the separately read names variable read from the Python pickle batches.meta file of string names like automobile, bird, etc. Each of the five sets of 10,000 images and labels are vertically and horizontally stacked onto two NumPy arrays, data of size (50000,1024) and labels (5000,), respectively, and made available for training. Listing 12.2 gets the CIFAR-10 data ready for the Autoencoder.