Tensorflow in Deep Learning Lecture 2: Neural Network JAHANDAR JAHANIPOUR jjahanipour@uh.edu www.easy-tensorflow.com https://github.com/easy-tensorflow 30/04/2018 www.easy-tensorflow.com

Outline Neural Network TensorBoard Introduction to Neural Network Implementing a Neural Network in TensorFlow TensorBoard Visualize the network graph Write summary 30/04/2018 www.easy-tensorflow.com

Neural Network Biological Neuron: The basic computational unit of the brain. Mathematical model

Activation Functions nonlinearity: 30/04/2018 www.easy-tensorflow.com

Neural Network #parameters = 200x784 + 200 + 10x200 + 10 =159,010 30/04/2018 www.easy-tensorflow.com

Neural Network Why None? b = tf.get_variable('bias', initializer=tf.zeros(200)) W = tf.get_variable('weight', shape=(784, 200), initializer=tf.truncated_normal_initializer(mean=0.0, stddev=0.01)) X = tf.placeholder(tf.float32, shape=[None, 784], name='input') y = tf.placeholder(tf.float32, shape=[None, 10], name='label') Why None? 30/04/2018 www.easy-tensorflow.com

Neural Network In each Iteration: (? = batch size) ?x784 ?x200 ?x10 30/04/2018 www.easy-tensorflow.com

Neural Network In each Iteration: (batch size = 100) 100x784 100x200 30/04/2018 www.easy-tensorflow.com

Neural Network Test time: (10,000 test samples) 10,000x784 10,000x200 30/04/2018 www.easy-tensorflow.com

Hyper-parameters variables # Data Dimensions img_h = img_w = 28 # MNIST images are 28x28 img_size_flat = img_h * img_w # 28x28=784, the total number of pixels n_classes = 10 # Number of classes, one class per digit # Hyper-parameters learning_rate = 0.001 # The optimization initial learning rate epochs = 10 # Total number of training epochs batch_size = 100 # Training batch size display_freq = 100 # Frequency of displaying the training results # Network Configuration h1 = 200 # Number of units in the first hidden layer variables 30/04/2018 www.easy-tensorflow.com

Build Graph Lazy Programming # Placeholders for inputs (x), outputs(y) x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='X') y = tf.placeholder(tf.float32, shape=[None, n_classes], name='Y') # Hidden layer W1 = tf.get_variable('W_1', dtype=tf.float32, shape=[784, 200], initializer=tf.truncated_normal_initializer(stddev=0.01)) b1 = tf.get_variable('b_1', dtype=tf.float32, initializer=tf.constant(0., shape=[200], dtype=tf.float32)) wxb = tf.matmul(x, W1) + b1 fc_1 = tf.nn.relu(wxb) # Output layer W2 = tf.get_variable('W_2', dtype=tf.float32, shape=[200, 10], initializer=tf.truncated_normal_initializer(stddev=0.01)) b2 = tf.get_variable('b_2', dtype=tf.float32, initializer=tf.constant(0., shape=[10], dtype=tf.float32)) output_logits = tf.matmul(fc_1, W2) + b2 Lazy Programming 30/04/2018 www.easy-tensorflow.com

Build Graph # Create the network graph x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='X') y = tf.placeholder(tf.float32, shape=[None, n_classes], name='Y') fc1 = fc_layer(x, h1, 'FC1', use_relu=True) output_logits = fc_layer(fc1, n_classes, 'OUT', use_relu=False) [?,784] 200 def weight_variable(name, shape): initer = tf.truncated_normal_initializer(stddev=0.01) return tf.get_variable('W_' + name, dtype=tf.float32, shape=shape, initializer=initer) def fc_layer(x, num_units, name, use_relu=True): in_dim = x.get_shape()[1] W = weight_variable(name, shape=[in_dim, num_units]) b = bias_variable(name, [num_units]) layer = tf.matmul(x, W) layer += b if use_relu: layer = tf.nn.relu(layer) return layer 784 [784,200] [200] [?,200] [?,200] def bias_variable(name, shape): initial = tf.constant(0., shape=shape, dtype=tf.float32) return tf.get_variable('b_' + name, dtype=tf.float32, initializer=initial) [?,200] 30/04/2018 www.easy-tensorflow.com

Create Optimization Operations # Define the loss function, optimizer, and accuracy loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=output_logits), name='loss') optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, name='Adam-op').minimize(loss) correct_prediction = tf.equal(tf.argmax(output_logits, 1), tf.argmax(y, 1), name='correct_pred') accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name='accuracy') # Network predictions cls_prediction = tf.argmax(output_logits, axis=1, name='predictions') # Initializing the variables init = tf.global_variables_initializer() Adam-op loss accuracy correct_pred OUT fc1 init Y X 30/04/2018 www.easy-tensorflow.com

Session # Launch the graph (session) with tf.Session() as sess: sess.run(init) # Number of training iterations in each epoch num_tr_iter = int(mnist.train.num_examples / batch_size) for epoch in range(epochs): print('Training epoch: {}'.format(epoch+1)) for iteration in range(num_tr_iter): batch_x, batch_y = mnist.train.next_batch(batch_size) # Run optimization op (backprop) feed_dict_batch = {x: batch_x, y: batch_y} sess.run(optimizer, feed_dict=feed_dict_batch) 30/04/2018 www.easy-tensorflow.com

Tensorboard is a flashlight for our Neural Net's black box. Visualize graph Write summaries Embedding Visualization 30/04/2018 www.easy-tensorflow.com

1. Visualize the graph Let’s clean up the graph ! Write data from TensorFlow to disc ! # Launch the graph (session) with tf.Session() as sess: train_writer = tf.summary.FileWriter(logdir='logs', graph=sess.graph) Then, in the command line type: >> tensorboard --logdir=logs Let’s clean up the graph ! Name scope / Variable scope Good luck with understating it ! 30/04/2018 www.easy-tensorflow.com

1. Visualize the graph with tf.name_scope("Input"): x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='X') y = tf.placeholder(tf.float32, shape=[None, n_classes], name='Y') fc1 = fc_layer(x, h1, 'FC1', use_relu=True) output_logits = fc_layer(fc1, n_classes, 'OUT', use_relu=False) with tf.name_scope("Train"): # Define the loss function, optimizer, and accuracy with tf.name_scope("Loss"): loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=output_logits), name='loss') with tf.name_scope("Optimizer"): optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, name='Adam-op').minimize(loss) with tf.name_scope("Accuracy"): correct_prediction = tf.equal(tf.argmax(output_logits, 1), tf.argmax(y, 1), name='correct_pred') accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name='accuracy') with tf.name_scope("Prediction"): # Network predictions cls_prediction = tf.argmax(output_logits, axis=1, name='predictions') def fc_layer(x, num_units, name, use_relu=True): with tf.variable_scope(name): in_dim = x.get_shape()[1] W = weight_variable(name, shape=[in_dim, num_units]) b = bias_variable(name, [num_units]) layer = tf.matmul(x, W) layer += b if use_relu: layer = tf.nn.relu(layer) return layer 30/04/2018 www.easy-tensorflow.com

2. Write Summaries tf.summary.scalar with tf.variable_scope('Train'): with tf.variable_scope('Loss'): loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=output_logits), name='loss') tf.summary.scalar('loss', loss) with tf.variable_scope('Optimizer'): optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, name='Adam-op').minimize(loss) with tf.variable_scope('Accuracy'): correct_prediction = tf.equal(tf.argmax(output_logits, 1), tf.argmax(y, 1), name='correct_pred') accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name='accuracy') tf.summary.scalar('Accuracy', accuracy) with tf.variable_scope('Prediction'): # Network predictions cls_prediction = tf.argmax(output_logits, axis=1, name='predictions') 30/04/2018 www.easy-tensorflow.com

2. Write Summaries tf.summary.image with tf.name_scope("Input"): x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='X') tf.summary.image('input', tf.reshape(x, (-1, img_h, img_w, 1)), 3) y = tf.placeholder(tf.float32, shape=[None, n_classes], name='Y') 30/04/2018 www.easy-tensorflow.com

2. Write Summaries tf.summary.histogram def fc_layer(x, num_units, name, use_relu=True): with tf.variable_scope(name): in_dim = x.get_shape()[1] W = weight_variable(name, shape=[in_dim, num_units]) b = bias_variable(name, [num_units]) tf.summary.histogram('W', W) tf.summary.histogram('b', b) layer = tf.matmul(x, W) layer += b if use_relu: layer = tf.nn.relu(layer) return layer 30/04/2018 www.easy-tensorflow.com

2. Write Summaries Lots of summary ops ! Do we have to run them individually? # Merge all summaries merged = tf.summary.merge_all() with tf.Session() as sess: train_writer = tf.summary.FileWriter(logdir='logs', graph=sess.graph) sess.run(init) # Number of training iterations in each epoch num_tr_iter = int(mnist.train.num_examples / batch_size) for epoch in range(epochs): for iteration in range(num_tr_iter): batch_x, batch_y = mnist.train.next_batch(batch_size) # Run optimization op (backprop) feed_dict_batch = {x: batch_x, y: batch_y} _, summary_tr = sess.run([optimizer, merged], feed_dict=feed_dict_batch) train_writer.add_summary(summary_tr, global_step) 30/04/2018 www.easy-tensorflow.com

2. Write Summaries You can run your network with several hyper-parameters and save them in subfolders: Let’s change the hidden layer size and see how the parameters change ! From “logs” parent folder: >> tensorboard --logdir=logs 30/04/2018 www.easy-tensorflow.com