1. Tensorflow in Deep Learning
Lecture 3: Auto-Encoder
JAHANDAR JAHANIPOUR
01/05/2018

2. Outline Recap: Neural Network Autoencoder TensorBoard
Denoising Autoencoder
Visualize Activation
TensorBoard
Embedding Projector
01/05/2018

3. Neural Network 28
𝓛𝒐𝒔𝒔= 𝟏 𝑵 𝒏=𝟏 𝑵 𝑷 𝒏 − 𝑳 𝒏 𝟐
Supervised Learning: We use labels to train the model
01/05/2018

4. Autoencoder Unsupervised Learning: No Label
Create a new representation in Latent Space
Find the most important features of the data
Reconstruct the input data: 𝑥 =𝑓 𝑥 ≃𝑥
ℒ 𝑥 , 𝑥 = 1 𝑁 𝑛=1 𝑁 𝑥 −𝑥 2
Types:
Denoising Autoencoder
Sparse Autoencoder
Latent Space
Image credit:
Variational Autoencoder …
01/05/2018

5. Autoencoder Graph: 𝓛 𝒙 , 𝒙 = 𝟏 𝑵 𝒏=𝟏 𝑵 𝒙 −𝒙 𝟐
𝓛 𝒙 , 𝒙 = 𝟏 𝑵 𝒏=𝟏 𝑵 𝒙 −𝒙 𝟐
Graph:
with tf.name_scope('Input'): x = tf.placeholder(tf.float32, shape=[None, 784], name='X') fc1 = fc_layer(x, 100, 'Hidden_layer', use_relu=True out = fc_layer(fc1, 784, 'Output_layer', use_relu=False) with tf.name_scope('Loss'): loss = tf.reduce_mean(tf.losses.mean_squared_error(x, out), name='loss') optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, name='Adam-op').minimize(loss)
01/05/2018

6. Denoising Autoencoder
(noisy)
𝓛 𝒙 , 𝒙 = 𝟏 𝑵 𝒏=𝟏 𝑵 𝒙 −𝒙 𝟐
Graph:
with tf.name_scope('Input'): x_original = tf.placeholder(tf.float32, shape=[None, 784], name='X_original')
x_noisy = tf.placeholder(tf.float32, shape=[None, 784], name='X_noisy')
fc1 = fc_layer(x_noisy, 100, 'Hidden_layer', use_relu=True) out = fc_layer(fc1, 784, 'Output_layer', use_relu=False) with tf.name_scope('Train'): loss = tf.reduce_mean(tf.losses.mean_squared_error(x_origianl, out), name='loss') optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, name='Adam-op').minimize(loss)
01/05/2018

7. Denoising Autoencoder
(noisy)
𝓛 𝒙 , 𝒙 = 𝟏 𝑵 𝒏=𝟏 𝑵 𝒙 −𝒙 𝟐
Session:
with tf.Session() as sess: for epoch in range(epochs): for iteration in range(num_tr_iter): batch_x, _ = mnist.train.next_batch(batch_size) batch_x_noisy = batch_x + noise_level * np.random.normal(loc=0.0, scale=1.0, size=batch_x.shape) # Run optimization op (backprop) feed_dict_batch = {x_original: batch_x, x_noisy: batch_x_noisy} _, loss_batch = sess.run([optimizer, loss], feed_dict=feed_dict_batch)
01/05/2018

8. Visualize Activation Understand what Autoencoder learned
𝑎 𝑖 =𝑓 𝑗= 𝑊 𝑖𝑗 1 𝑥 𝑗 + 𝑏 𝑖
Which input image maximizes the activation of node in hidden layer
𝑥 𝑖 = 𝑊 𝑖𝑗 𝑗= 𝑊 𝑖𝑗 1 2
h_active = W1 / tf.sqrt(tf.reduce_sum(tf.square(W1), axis=0))
𝑥 1 =?
Image credit:
𝑥 2 =?
𝑥 3 =?
𝑥 4 =?
𝑥 5 =?
𝑥 6 =?
01/05/2018

9. Outline Recap: Neural Network Autoencoder TensorBoard
Denoising Autoencoder
Visualize Activation
TensorBoard
Embedding Projector
01/05/2018

10. Embedding Projector
Passing the test set and visualizing the tensor of activation of hidden layer nodes in test set:
𝑒𝑎𝑐ℎ 𝑠𝑎𝑚𝑝𝑙𝑒∈ ℛ 200
How to visualize the ℛ 200 space?
Dimension Reduction to ℛ 3 :
PCA
t-SNE
Image credit:
01/05/2018

11. Embedding Projector Import Create an embedding variable
from tensorflow.contrib.tensorboard.plugins import projector
Create an embedding variable
# Load the test set x_test = mnist.test.images y_test = mnist.test.labels # Initialize the embedding variable with the shape of our desired tensor tensor_shape = (x_test.shape[0] , fc1.get_shape()[1].value) # [test_set , h1] embedding_var = tf.Variable(tf.zeros(tensor_shape), name='fc1_embedding') # assign the tensor that we want to visualize to the embedding variable embedding_assign = embedding_var.assign(fc1)
Create an embedding Projector
# Create a config object to write the configuration parameters config = projector.ProjectorConfig() # Add embedding variable embedding = config.embeddings.add() embedding.tensor_name = embedding_var.name
01/05/2018

12. Embedding Projector Add metadata information:
# Reshape images from vector to matrix x_test_images = np.reshape(np.array(x_test), (-1, img_w, img_h)) # Reshape labels from one-hot-encode to index x_test_labels = np.argmax(y_test, axis=1) write_sprite_image(os.path.join(logs_path, 'sprite_images.png'), x_test_images) write_metadata(os.path.join(logs_path, 'metadata.tsv'), x_test_labels)
# Link this tensor to its metadata file (e.g. labels) embedding.metadata_path = 'metadata.tsv' # Specify where you find the sprite embedding.sprite.image_path = 'sprite_images.png' embedding.sprite.single_image_dim.extend([img_w, img_h]) # Write a projector_config.pbtxt in the logs_path # TensorBoard will read this file during startup projector.visualize_embeddings(train_writer, config)
01/05/2018

13. Embedding Projector In Session:
# Run session to evaluate the tensor x_test_fc1 = sess.run(embedding_assign, feed_dict={x: x_test}) # Save the tensor in model.ckpt file saver = tf.train.Saver() saver.save(sess, os.path.join(logs_path, "model.ckpt"), global_step)
Finally, run TensorBoard in terminal (from the parent folder of the folder the log is saved):
tensorboard --logdir=logs
01/05/2018