Deep learning Introduction Classes of Deep Learning Networks Unsupervised or Generative Learning Supervised Learning Hybrid Networks References Images obtained from multiple sources, including Wikipedia, articles, and blogs from the internet.

Introduction Definition (from reference [3]). What is machine learning? Definition (from reference [3]). A sub-field of machine learning that is based on learning several levels of representations, corresponding to a hierarchy of features or factors or concepts, where higher level concepts are defined from lower-level ones, and the same lower-level concepts can help to define many higher-level concepts. (Wikipedia on “Deep Learning”)

Introduction Deep learning is popular because of the following: Useful in both supervised and unsupervised leaning. Can be implemented with modern hardware Advances in machine learning provide the tools to train these architectures. What is machine learning? Figure obtained from reference [3]. The goal is to learn with deep architectures in signal and info. processing.

Introduction General Setting Deep Learning was too difficult to do because of the credit-assignment problem. Recent discoveries have tackled this barrier making it possible to learn deep network architectures. Who to blame? What is machine learning?

Introduction We want to capture compact, high-level representations in an efficient and iterative manner. Learning takes place at several levels of representations. Think about a hierarchy of concepts of increasing complexity. Low levels concepts are the foundation for high level concepts. What is machine learning?

Deep learning Introduction Classes of Deep Learning Networks Unsupervised or Generative Learning Supervised Learning Hybrid Networks References

Classes of Deep Learning Networks There are three basic types on Deep Networks: Deep Networks for unsupervised or generative learning. Capture high order correlations of the data (no class labels) Deep Networks for Supervised Learning Model the posterior distribution of the target variable for classification purposes (Discriminative Deep Networks). Hybrid Deep Networks Combine the methods above.

Deep Networks for Unsupervised Learning There are no class labels during the learning process. There are many types of generative or unsupervised deep networks. Energy-based deep networks are very popular. Example: Deep Auto Encoder.

Deep Learning Auto Encoder Auto Encoder

Deep Networks for Unsupervised Learning No. of output features = No input features Decoder Auto Encoder Encoder Intermediate nodes encode the original data.

Deep Networks for Unsupervised Learning Network is trained using backpropagation. Output nodes replicate the input nodes. Originally used to re-construct noisy signals. First layer of weights “encode” the signal. Second layer of weights “decode” the signal The intermediate layer contains a “new” feature representation. Auto Encoder

Deep Networks for Unsupervised Learning New feature representation Auto Encoder

Deep Networks for Unsupervised Learning “Deep” Auto Encoder Using a Step-Wise Mechanism Key idea: Pre-train each layer as an auto-encoder.

An Example of Deep Learning Learn a “concept” (sedimentary rocks) from many images until a high-level representation is achieved.

An Example of Deep Learning Learn a hierarchy of abstract concepts using deep learning. Global properties Deep Learning Local properties

Deep Networks for Unsupervised Learning Deep Autoencoders to Extract Speech Features

Introduction History of Neural Networks What is machine learning? Figure obtained from reference [3].

Introduction Original Idea appeared in the following reference: A Fast Learning Algorithm for Deep Belief Nets by G. Hinton, S. Osindero, and Y. W. The. IEEE Transactions on Neural Computation Vol. 18 pp. 1527-1554, 2006. Geoffrey Hinton Researcher at Google Invented “backpropagation” What is machine learning?

Deep learning Introduction Classes of Deep Learning Networks Unsupervised or Generative Learning Supervised Learning Hybrid Networks References

Deep Networks for Supervised Learning The network can be trained directly using backpropagation if the right activation function is used. ReLU: O.K. Sigmoid: Wrong What is machine learning?

Deep Networks for Supervised Learning Rectified Linear Unit: ReLU The function is defined as follows: f(x)= max(0,x) It increases the nonlinear properties of the decision function and of the overall network without affecting the receptive fields of the convolution layer. In sigmoid function units can saturate. What is machine learning?

Commercial Planes, Military Planes Deep Networks for Unsupervised Learning The idea is to disentangle factors of variation and to attain high level representations. Commercial Planes, Military Planes Engine, Main Fuselage Small Object Parts What is machine learning? Edges and Contours Pixel Information

Deep Networks for Supervised Learning Deep Stacking Network Combines deep learning in Neural Networks with Stacking. + Stacking

Deep Networks for Supervised Learning Deep Stacking Network General Architecture Networks are trained separately but in sequence. Output layer is input to a new network Original input can be re-used as input

Deep Learning for Supervised Learning Convolutional Neural Networks Primarily used for image analysis. Inspired by animal visual cortex Neurons respond to input from near neurons in what is called the receptive field.

Deep Learning for Supervised Learning Design of a Convolutional Neural Network CNN A CNN has input, output and hidden units. Hidden units can be of 3 types: Convolutional Pooling Fully Connected output Convolutional hidden Pooling Fully Connected input

Deep Learning for Supervised Learning Design of a Convolutional Neural Network CNN Why are CNNs important when dealing with images? Assume an image of size 500x500 pixels and each pixel can have 3 color channels. 500 pixels 1 pixel w/ 3 color channels 500 pixels A fully connected NN needs a hidden layer with 500x500x3 weights = 750,000 weights!

Deep Learning for Supervised Learning Design of a Convolutional Neural Network CNN Why are CNNs important when dealing with images? Instead a CNN with have local set of weights only. Each neuron will be connected to a few close by neurons only (idea of receptive field)

Deep Learning for Supervised Learning Design of a Convolutional Neural Network CNN First difference with traditional NN is that neurons are arranged in 3 dimensions.

Deep Learning for Supervised Learning Design of a Convolutional Neural Network CNN

Deep Learning for Supervised Learning Convolutional Neural Networks Local Weight Update Implies a sparse representation

Deep Learning for Supervised Learning Convolutional Neural Networks Convolution Operation We need to learn the kernel K and share those parameters across the entire image.

Deep Learning for Supervised Learning Convolutional Neural Networks Vertical and horizontal filters:

Deep Learning for Supervised Learning Convolutional Neural Networks

Deep Learning for Supervised Learning Convolutional Neural Networks Convolution Operation We can learn rotated objects using convolution.

Deep Learning for Supervised Learning Design of a Convolutional Neural Network CNN Layers alternate between convolutional layers and pooling layers:

Deep Learning for Supervised Learning Design of a Convolutional Neural Network CNN Pooling aggressively reduces the dimensionality of the feature space. The idea is as follows: We partition the image into a set of non-overlapping rectangles. For each region we simply output the maximum value of that region (set of pixels). This is called “Max Pooling”.

Deep Learning for Supervised Learning Design of a Convolutional Neural Network CNN Full convolutional neural network : Apply convolution, pooling (or subsampling) iteratively. Finally apply fully connected neural network:

Deep learning Introduction Classes of Deep Learning Networks Unsupervised or Generative Learning Supervised Learning Hybrid Networks References

Deep Networks for Hybrid Learning Pre-trained Deep Neural Network

Deep learning Introduction Classes of Deep Learning Networks Unsupervised or Generative Learning Supervised Learning Hybrid Networks References

References [1] A Fast Learning Algorithm for Deep Belief Nets by G. Hinton, S. Osindero, and Y. W. The. IEEE Transactions on Neural Computation Vol. 18 pp. 1527-1554, 2006. [2] Representation Learning: A Review and New Perspectives by Y. Bengio, A. Courville, and P. Vincent. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 35 No. 8, pp. 1798 – 1828, 2013. [3] Deep Learning: Methods and Applications Li Deng and Dong Yu, NOW PUBLISHERS, 2014 [4] Binary Coding of Speech Spectrograms Using a Deep Autoencoder. L. Deng, M. Seltzer, D. Yu, A. Acer1, A. Mohamed, and G. Hinton. Interspeech, 2010.

References [5] Deep Learning Neural Networks: Design and Case Studies by Daniel Graupe, World Scientific Publishing 2016. [6] Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville, MIT Press, 2016. [7] Deep Learning: A Practitioner’s Approach by Josh Patterson and Adam Gibson, O’Reilly Media Inc., 2017. [8] Fundamentals of Deep Learning by Nikhil Buduma, O’Reilly Media Inc., 2017.