Tensorflow in Deep Learning

Slides:



Advertisements
Similar presentations
ImageNet Classification with Deep Convolutional Neural Networks
Advertisements

Handwritten Character Recognition Using Artificial Neural Networks Shimie Atkins & Daniel Marco Supervisor: Johanan Erez Technion - Israel Institute of.
Lecture 3: CNN: Back-propagation
Overview of Back Propagation Algorithm
Convolutional Neural Networks for Image Processing with Applications in Mobile Robotics By, Sruthi Moola.
Object Recognition Tutorial Beatrice van Eden - Part time PhD Student at the University of the Witwatersrand. - Fulltime employee of the Council for Scientific.
Convolutional Neural Network
Deep Learning Overview Sources: workshop-tutorial-final.pdf
Assignment 4: Deep Convolutional Neural Networks
Lecture 3a Analysis of training of NN
Reinforcement Learning
Demo.
CS 6501: 3D Reconstruction and Understanding Convolutional Neural Networks Connelly Barnes.
Artificial Neural Networks
Data Mining, Neural Network and Genetic Programming
Data Mining, Neural Network and Genetic Programming
Computer Science and Engineering, Seoul National University
Convolutional Neural Fabrics by Shreyas Saxena, Jakob Verbeek
Applications of Deep Learning and how to get started with implementation of deep learning Presentation By : Manaswi Advisor : Dr.Chinmay.
Principal Solutions AWS Deep Learning
Deep Learning Libraries
Intro to NLP and Deep Learning
Lecture 25: Backprop and convnets
Mini Presentations - part 2
Lecture 5 Smaller Network: CNN
Neural Networks 2 CS446 Machine Learning.
Convolution Neural Networks
Convolutional Networks
CS6890 Deep Learning Weizhen Cai
Non-linear classifiers Neural networks
Deep Learning Convoluted Neural Networks Part 2 11/13/
Tensorflow in Deep Learning
Computer Vision James Hays
Tensorflow in Deep Learning
Introduction to Neural Networks
Introduction to Tensorflow
Convolutional Neural Network
CSC 578 Neural Networks and Deep Learning
MXNet Internals Cyrus M. Vahid, Principal Solutions Architect,
Convolutional Neural Networks
Construct a Convolutional Neural Network with Python
Hairong Qi, Gonzalez Family Professor
Very Deep Convolutional Networks for Large-Scale Image Recognition
Smart Robots, Drones, IoT
Convolutional neural networks Abin - Roozgard.
CSC 578 Neural Networks and Deep Learning
Deep Learning and Mixed Integer Optimization
On Convolutional Neural Network
Vinit Shah, Joseph Picone and Iyad Obeid
Forward and Backward Max Pooling
Analysis of Trained CNN (Receptive Field & Weights of Network)
Debugging Dataflow Graphs using TensorFlow Debugger.
Coding neural networks: A gentle Introduction to keras
Convolutional Neural Networks
Neural Networks.
Mihir Patel and Nikhil Sardana
实习生汇报 ——北邮 张安迪.
Image Classification & Training of Neural Networks
CSC 578 Neural Networks and Deep Learning
Lecture 7 Spatial filtering.
Deep Learning Libraries
CS295: Modern Systems: Application Case Study Neural Network Accelerator Sang-Woo Jun Spring 2019 Many slides adapted from Hyoukjun Kwon‘s Gatech “Designing.
VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION
Image recognition.
Debasis Bhattacharya, JD, DBA University of Hawaii Maui College
Example of training and deployment of deep convolutional neural networks. Example of training and deployment of deep convolutional neural networks. During.
CSC 578 Neural Networks and Deep Learning
CSC 578 Neural Networks and Deep Learning
Deep Learning with TensorFlow
Tensorflow in Deep Learning
Presentation transcript:

Tensorflow in Deep Learning Lecture 4: Convolutional Neural Network (CNN) JAHANDAR JAHANIPOUR jjahanipour@uh.edu www.easy-tensorflow.com https://github.com/easy-tensorflow 01/05/2018 www.easy-tensorflow.com

Outline Feed-Forward Neural Network shortcomings Convolutional Neural Networks How does a CNN work Convolution layer Pooling layer Implementing CNN using TensorFlow Save and Restore the Network 01/05/2018 www.easy-tensorflow.com

Feed Forward Neural Network (NN) 01/05/2018 www.easy-tensorflow.com

Neural Network Problems: Doesn’t use data structure! Translation invariance 01/05/2018 www.easy-tensorflow.com

Neural Network Problems: Doesn’t use data structure! Solution: Weight sharing CNN: Neural Net that shares it’s parameters across space W W 01/05/2018 www.easy-tensorflow.com

Neural Network Problems: Doesn’t scale well to full images 784 Units 500 Units #parameters = 784 x 500 + 500 = 392K !!! 01/05/2018 www.easy-tensorflow.com

Neural Network Problems: Doesn’t scale well to full images Solution: Weight sharing + 3D volume of neurons Sharing the same set of weights (and biases) 01/05/2018 www.easy-tensorflow.com

Layers Used to Build CNN 01/05/2018 www.easy-tensorflow.com

Convolution Layer What is convolution? A function derived from two given functions by integration that expresses how the shape of one is modified by the other. Continues: 𝑓∗𝑔 𝑡 = 𝑓 𝜏 𝑔 𝑡−𝜏 𝑑𝜏 𝐷𝑖𝑠𝑐𝑟𝑒𝑡𝑒: 𝑓∗𝑔 𝑛 = 𝑓 𝑚 𝑔[𝑛−𝑚] 1. slide 2. multiply 3. integrate (i.e. sum) 01/05/2018 www.easy-tensorflow.com

Convolution Layer - Spatial dimensions: 32x32 - Depth: 3 Feature maps (R+G+B) 01/05/2018 www.easy-tensorflow.com

Convolution Layer (Filter = Kernel = Patch) Convolve the filter with the image i.e. “slide over the image spatially, Computing dot products, And sum over all” 01/05/2018 www.easy-tensorflow.com

Convolution Layer 01/05/2018 www.easy-tensorflow.com

Convolution Layer 01/05/2018 www.easy-tensorflow.com

Convolution Layer 01/05/2018 www.easy-tensorflow.com

Convolution Layer 01/05/2018 www.easy-tensorflow.com

Convolution Layer 01/05/2018 www.easy-tensorflow.com

Convolution Layer 01/05/2018 www.easy-tensorflow.com

Convolution Layer o=(i-K)/s +1 => Output size: o = 3 A closer look at spatial dimensions: Input size: i = 7 Filter size: K=3 Stride: s=2 => Output size: o = 3 o=(i-K)/s +1 01/05/2018 www.easy-tensorflow.com

Convolution Layer o=(i-K)/s +1 = 2.33 !!! o=(i-K)/s +1 =? A closer look at spatial dimensions: Input size: i = 7 Filter size: K=3 Stride: s=3 o=(i-K)/s +1 = 2.33  !!! o=(i-K)/s +1 =? 01/05/2018 www.easy-tensorflow.com

Convolution Layer o=(i+2p-K)/s +1 Zero-padding: A closer look at spatial dimensions: o=(i+2p-K)/s +1 Zero-padding: Input size: i = 7 Filter size: K=3 Stride: s=3 Zero-pad: p=1 => Output size: o = 3 01/05/2018 www.easy-tensorflow.com

Convolution Layer A closer look at spatial dimensions: Valid padding: don’t go past the edges Same padding: go off the edge (with s=1), pad with zeros in such a way that output size = input size K=3 => zero pad with 1 K=5 => zero pad with 2 K=7 => zero o=(i+2p-K)/s +1  => i=(i+2p-K)+1  =>  p=(K-1)/2 01/05/2018 www.easy-tensorflow.com

Convolution in TensorFlow Computes a 2-D convolution given 4-D input and filter tensors input: tensor of shape [batch_size, in_height, in_width, in_channels] filter: tensor of shape [filter_height, filter_width, in_channels, out_channels] tf.nn.conv2d(     input,     filter,     strides,     padding,     use_cudnn_on_gpu=True,     data_format='NHWC',     dilations=[1, 1, 1, 1],     name=None ) 01/05/2018 www.easy-tensorflow.com

Helper Function for Convolution Layer def conv_layer(x, filter_size, num_filters, stride, name): with tf.variable_scope(name): num_in_channel = x.get_shape().as_list()[-1] shape = [filter_size, filter_size, num_in_channel, num_filters] W = weight_variable(shape=shape) tf.summary.histogram('weight', W) b = bias_variable(shape=[num_filters]) tf.summary.histogram('bias', b) layer = tf.nn.conv2d(x, W, strides=[1, stride, stride, 1], padding="SAME") layer += b return tf.nn.relu(layer) 01/05/2018 www.easy-tensorflow.com

Pooling Layer To reduce the spatial dimension of feature maps reduce the amount of parameters 01/05/2018 www.easy-tensorflow.com

Max Pooling F = Filter Size S = Stride Common Settings: F = 2, S = 2 01/05/2018 www.easy-tensorflow.com

Max Pooling o=(i-F)/s +1 Input size: i = 5 Filter size: F=3 Stride:        s=1 => Output size: o = 3 01/05/2018 www.easy-tensorflow.com

Average Pooling 01/05/2018 www.easy-tensorflow.com

Helper Function for Pooling Layer def max_pool(x, ksize, stride, name): return tf.nn.max_pool(x, ksize=[1, ksize, ksize, 1], strides=[1, stride, stride, 1], padding="SAME", name=name) 01/05/2018 www.easy-tensorflow.com

Helper Function for Flatten Layer and Fully Connected Layer def fc_layer(x, num_units, name, use_relu=True): with tf.variable_scope(name): in_dim = x.get_shape()[1] W = weight_variable(shape=[in_dim, num_units]) tf.summary.histogram('weight', W) b = bias_variable(shape=[num_units]) tf.summary.histogram('bias', b) layer = tf.matmul(x, W) layer += b if use_relu: layer = tf.nn.relu(layer) return layer def flatten_layer(layer): with tf.variable_scope('Flatten_layer'): layer_shape = layer.get_shape() num_features = layer_shape[1:4].num_elements() layer_flat = tf.reshape(layer, [-1, num_features]) return layer_flat 01/05/2018 www.easy-tensorflow.com

Remarks on Implementation # hyper-parameters learning_rate = 0.001# The optimization learning rate epochs = 10 # Total number of training epochs batch_size = 100 # Training batch size # Network Parameters # We know that MNIST images are 28 pixels in each dimension. img_h = img_w = 28 # Images are stored in one-dimensional arrays of this length. img_size_flat = img_h * img_w # Number of classes, one class for each of 10 digits. n_classes = 10 01/05/2018 www.easy-tensorflow.com

Remarks on Implementation Feed-forward network: # Placeholders for inputs (x), outputs(y) with tf.name_scope('input'): x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='X') y = tf.placeholder(tf.float32, shape=[None, n_classes], name='Y') CNN # Placeholders for inputs (x), outputs(y) with tf.name_scope('input'): x = tf.placeholder(tf.float32, shape=[None, img_h, img_w, n_channels], name='X') y = tf.placeholder(tf.float32, shape=[None, n_classes], name='Y') 01/05/2018 www.easy-tensorflow.com

saver = tf.train.Saver() Save and Restore Model Process of training a network is time consuming TensorFlow has an easy solution to save and restore your model: In the graph: saver = tf.train.Saver() 01/05/2018 www.easy-tensorflow.com

Save and Restore Model In the session: Save In the session: (usually after training is done) saver.save(sess, model_path + model_name, global_step) Model is loaded with the TensorFlow names ! var_python_name = tf.get_variable("var_tf_name", shape=...) Restore in the session: (usually before running the test functions) saver.restore(sess, tf.train.latest_checkpoint(model_path)) Variable is saved with this name in the file 01/05/2018 www.easy-tensorflow.com

https://github.com/easy-tensorflow/easy-tensorflow If you found this workshop interesting: Please consider giving us star on GitHub and Follow us for more tutorials https://github.com/easy-tensorflow/easy-tensorflow Feel free to leave comments on the tutorials, if you have any questions, suggestions … http://easy-tensorflow.com/ 01/05/2018 www.easy-tensorflow.com