1. Tensorflow in Deep Learning
Lecture 2: Neural Network
JAHANDAR JAHANIPOUR
30/04/2018

2. Outline Neural Network TensorBoard Introduction to Neural Network
Implementing a Neural Network in TensorFlow
TensorBoard
Visualize the network graph
Write summary
30/04/2018

3. Neural Network Biological Neuron:
The basic computational unit of the brain.
Mathematical model

4. Activation Functions
nonlinearity:
30/04/2018

5. Neural Network #parameters = 200x784 + 200 + 10x200 + 10 =159,010
30/04/2018

6. Neural Network Why None?
b = tf.get_variable('bias', initializer=tf.zeros(200)) W = tf.get_variable('weight', shape=(784, 200),
initializer=tf.truncated_normal_initializer(mean=0.0, stddev=0.01))
X = tf.placeholder(tf.float32, shape=[None, 784], name='input') y = tf.placeholder(tf.float32, shape=[None, 10], name='label')
Why None?
30/04/2018

7. Neural Network In each Iteration: (? = batch size) ?x784 ?x200 ?x10
30/04/2018

8. Neural Network In each Iteration: (batch size = 100) 100x784 100x200
30/04/2018

9. Neural Network Test time: (10,000 test samples) 10,000x784 10,000x200
30/04/2018

10. Hyper-parameters variables
# Data Dimensions img_h = img_w = # MNIST images are 28x28 img_size_flat = img_h * img_w # 28x28=784, the total number of pixels n_classes = # Number of classes, one class per digit # Hyper-parameters learning_rate = # The optimization initial learning rate epochs = # Total number of training epochs batch_size = # Training batch size display_freq = # Frequency of displaying the training results # Network Configuration h1 = # Number of units in the first hidden layer
variables
30/04/2018

11. Build Graph Lazy Programming
# Placeholders for inputs (x), outputs(y) x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='X') y = tf.placeholder(tf.float32, shape=[None, n_classes], name='Y')
# Hidden layer W1 = tf.get_variable('W_1', dtype=tf.float32, shape=[784, 200], initializer=tf.truncated_normal_initializer(stddev=0.01)) b1 = tf.get_variable('b_1', dtype=tf.float32, initializer=tf.constant(0., shape=[200], dtype=tf.float32)) wxb = tf.matmul(x, W1) + b1 fc_1 = tf.nn.relu(wxb)
# Output layer W2 = tf.get_variable('W_2', dtype=tf.float32, shape=[200, 10], initializer=tf.truncated_normal_initializer(stddev=0.01)) b2 = tf.get_variable('b_2', dtype=tf.float32, initializer=tf.constant(0., shape=[10], dtype=tf.float32)) output_logits = tf.matmul(fc_1, W2) + b2
Lazy Programming
30/04/2018

12. Build Graph
# Create the network graph x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='X') y = tf.placeholder(tf.float32, shape=[None, n_classes], name='Y') fc1 = fc_layer(x, h1, 'FC1', use_relu=True) output_logits = fc_layer(fc1, n_classes, 'OUT', use_relu=False)
[?,784]
200
def weight_variable(name, shape): initer = tf.truncated_normal_initializer(stddev=0.01) return tf.get_variable('W_' + name, dtype=tf.float32, shape=shape, initializer=initer)
def fc_layer(x, num_units, name, use_relu=True): in_dim = x.get_shape()[1] W = weight_variable(name,
shape=[in_dim, num_units]) b = bias_variable(name, [num_units]) layer = tf.matmul(x, W) layer += b if use_relu: layer = tf.nn.relu(layer) return layer
784
[784,200]
[200]
[?,200]
[?,200]
def bias_variable(name, shape): initial = tf.constant(0., shape=shape, dtype=tf.float32) return tf.get_variable('b_' + name, dtype=tf.float32, initializer=initial)
[?,200]
30/04/2018

13. Create Optimization Operations
# Define the loss function, optimizer, and accuracy loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=output_logits), name='loss') optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, name='Adam-op').minimize(loss) correct_prediction = tf.equal(tf.argmax(output_logits, 1), tf.argmax(y, 1), name='correct_pred') accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name='accuracy') # Network predictions cls_prediction = tf.argmax(output_logits, axis=1, name='predictions') # Initializing the variables init = tf.global_variables_initializer()
Adam-op
loss
accuracy
correct_pred
OUT
fc1
init Y X
30/04/2018

14. Session
# Launch the graph (session) with tf.Session() as sess: sess.run(init) # Number of training iterations in each epoch num_tr_iter = int(mnist.train.num_examples / batch_size) for epoch in range(epochs): print('Training epoch: {}'.format(epoch+1)) for iteration in range(num_tr_iter): batch_x, batch_y = mnist.train.next_batch(batch_size) # Run optimization op (backprop) feed_dict_batch = {x: batch_x, y: batch_y} sess.run(optimizer, feed_dict=feed_dict_batch)
30/04/2018

15. Tensorboard is a flashlight for our Neural Net's black box.
Visualize graph
Write summaries
Embedding Visualization
30/04/2018

16. 1. Visualize the graph Let’s clean up the graph !
Write data from TensorFlow to disc ! # Launch the graph (session) with tf.Session() as sess: train_writer = tf.summary.FileWriter(logdir='logs', graph=sess.graph)
Then, in the command line type: >> tensorboard --logdir=logs
Let’s clean up the graph !
Name scope / Variable scope
Good luck with understating it !
30/04/2018

17. 1. Visualize the graph
with tf.name_scope("Input"): x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='X') y = tf.placeholder(tf.float32, shape=[None, n_classes], name='Y') fc1 = fc_layer(x, h1, 'FC1', use_relu=True) output_logits = fc_layer(fc1, n_classes, 'OUT', use_relu=False) with tf.name_scope("Train"): # Define the loss function, optimizer, and accuracy with tf.name_scope("Loss"): loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=output_logits), name='loss') with tf.name_scope("Optimizer"): optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, name='Adam-op').minimize(loss) with tf.name_scope("Accuracy"): correct_prediction = tf.equal(tf.argmax(output_logits, 1), tf.argmax(y, 1), name='correct_pred') accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name='accuracy') with tf.name_scope("Prediction"): # Network predictions cls_prediction = tf.argmax(output_logits, axis=1, name='predictions')
def fc_layer(x, num_units, name, use_relu=True):
with tf.variable_scope(name): in_dim = x.get_shape()[1] W = weight_variable(name, shape=[in_dim, num_units]) b = bias_variable(name, [num_units]) layer = tf.matmul(x, W) layer += b if use_relu: layer = tf.nn.relu(layer) return layer
30/04/2018

18. 2. Write Summaries tf.summary.scalar
with tf.variable_scope('Train'): with tf.variable_scope('Loss'): loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=output_logits), name='loss') tf.summary.scalar('loss', loss) with tf.variable_scope('Optimizer'): optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, name='Adam-op').minimize(loss) with tf.variable_scope('Accuracy'): correct_prediction = tf.equal(tf.argmax(output_logits, 1), tf.argmax(y, 1), name='correct_pred') accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name='accuracy') tf.summary.scalar('Accuracy', accuracy) with tf.variable_scope('Prediction'): # Network predictions cls_prediction = tf.argmax(output_logits, axis=1, name='predictions')
30/04/2018

19. 2. Write Summaries tf.summary.image
with tf.name_scope("Input"): x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='X') tf.summary.image('input', tf.reshape(x, (-1, img_h, img_w, 1)), 3) y = tf.placeholder(tf.float32, shape=[None, n_classes], name='Y')
30/04/2018

20. 2. Write Summaries tf.summary.histogram
def fc_layer(x, num_units, name, use_relu=True):
with tf.variable_scope(name): in_dim = x.get_shape()[1] W = weight_variable(name, shape=[in_dim, num_units]) b = bias_variable(name, [num_units]) tf.summary.histogram('W', W)
tf.summary.histogram('b', b) layer = tf.matmul(x, W) layer += b if use_relu: layer = tf.nn.relu(layer) return layer
30/04/2018

21. 2. Write Summaries
Lots of summary ops ! Do we have to run them individually?
# Merge all summaries merged = tf.summary.merge_all()
with tf.Session() as sess: train_writer = tf.summary.FileWriter(logdir='logs', graph=sess.graph) sess.run(init) # Number of training iterations in each epoch num_tr_iter = int(mnist.train.num_examples / batch_size) for epoch in range(epochs): for iteration in range(num_tr_iter): batch_x, batch_y = mnist.train.next_batch(batch_size) # Run optimization op (backprop) feed_dict_batch = {x: batch_x, y: batch_y} _, summary_tr = sess.run([optimizer, merged], feed_dict=feed_dict_batch) train_writer.add_summary(summary_tr, global_step)
30/04/2018

22. 2. Write Summaries
You can run your network with several hyper-parameters and save them in subfolders:
Let’s change the hidden layer size and see how the parameters change !
From “logs” parent folder:
>> tensorboard --logdir=logs
30/04/2018