1. Deep learning Introduction Classes of Deep Learning Networks
Unsupervised or Generative Learning
Supervised Learning
Hybrid Networks
References
Images obtained from multiple sources, including Wikipedia, articles, and blogs from the internet.

2. Introduction Definition (from reference [3]).
What is machine learning?
Definition (from reference [3]).
A sub-field of machine learning that is based on learning several levels of representations, corresponding to a hierarchy of features or factors or concepts, where higher level concepts are defined from lower-level ones, and the same lower-level concepts can help to define many higher-level concepts.
(Wikipedia on “Deep Learning”)

3. Introduction Deep learning is popular because of the following:
Useful in both supervised and unsupervised leaning.
Can be implemented with modern hardware
Advances in machine learning provide the
tools to train these architectures.
What is machine learning?
Figure obtained from reference [3].
The goal is to learn with deep architectures in signal and info. processing.

4. Introduction General Setting
Deep Learning was too difficult to do because of the credit-assignment problem.
Recent discoveries have tackled this barrier making it possible to learn deep
network architectures.
Who to blame?
What is machine learning?

5. Introduction
We want to capture compact, high-level representations in an efficient and iterative manner.
Learning takes place at several levels
of representations.
Think about a hierarchy of concepts
of increasing complexity.
Low levels concepts are the foundation
for high level concepts.
What is machine learning?

6. Deep learning Introduction Classes of Deep Learning Networks
Unsupervised or Generative Learning
Supervised Learning
Hybrid Networks
References

7. Classes of Deep Learning Networks
There are three basic types on Deep Networks:
Deep Networks for unsupervised or generative learning.
Capture high order correlations of the data (no class labels)
Deep Networks for Supervised Learning
Model the posterior distribution of the target variable for classification purposes (Discriminative Deep Networks).
Hybrid Deep Networks
Combine the methods above.

8. Deep Networks for Unsupervised Learning
There are no class labels during the learning process.
There are many types of generative or unsupervised deep networks.
Energy-based deep networks are very popular.
Example: Deep Auto Encoder.

9. Deep Learning
Auto Encoder
Auto Encoder

10. Deep Networks for Unsupervised Learning
No. of output features = No input features
Decoder
Auto Encoder
Encoder
Intermediate nodes encode the original data.

11. Deep Networks for Unsupervised Learning
Network is trained using backpropagation.
Output nodes replicate the input nodes.
Originally used to re-construct noisy signals.
First layer of weights “encode” the signal.
Second layer of weights “decode” the signal
The intermediate layer contains a “new” feature representation.
Auto Encoder

12. Deep Networks for Unsupervised Learning
New feature
representation
Auto Encoder

13. Deep Networks for Unsupervised Learning
“Deep” Auto Encoder
Using a Step-Wise Mechanism
Key idea: Pre-train each layer as an auto-encoder.

14. An Example of Deep Learning
Learn a “concept” (sedimentary rocks) from many images until
a high-level representation is achieved.

15. An Example of Deep Learning
Learn a hierarchy of abstract concepts using deep learning.
Global properties
Deep Learning
Local properties

16. Deep Networks for Unsupervised Learning
Deep Autoencoders to Extract Speech Features

17. Introduction History of Neural Networks
What is machine learning?
Figure obtained from reference [3].

18. Introduction Original Idea appeared in the following reference:
A Fast Learning Algorithm for Deep Belief Nets
by G. Hinton, S. Osindero, and Y. W. The.
IEEE Transactions on Neural Computation Vol. 18 pp , 2006.
Geoffrey Hinton
Researcher at Google
Invented “backpropagation”
What is machine learning?

19. Deep learning Introduction Classes of Deep Learning Networks
Unsupervised or Generative Learning
Supervised Learning
Hybrid Networks
References

20. Deep Networks for Supervised Learning
The network can be trained directly using backpropagation if the right
activation function is used.
ReLU: O.K.
Sigmoid: Wrong
What is machine learning?

21. Deep Networks for Supervised Learning
Rectified Linear Unit: ReLU
The function is defined as follows:
f(x)= max(0,x)
It increases the nonlinear properties of the decision function and of the overall network without affecting the receptive fields of the convolution layer.
In sigmoid function units can saturate.
What is machine learning?

22. Commercial Planes, Military Planes
Deep Networks for Unsupervised Learning
The idea is to disentangle factors of variation and to
attain high level representations.
Commercial Planes, Military Planes
Engine,
Main Fuselage
Small
Object Parts
What is machine learning?
Edges and Contours
Pixel Information

23. Deep Networks for Supervised Learning
Deep Stacking Network
Combines deep learning in Neural Networks with Stacking.
+ Stacking

24. Deep Networks for Supervised Learning
Deep Stacking Network
General Architecture
Networks are trained separately but in
sequence.
Output layer is input to a new network
Original input can be re-used as input

25. Deep Learning for Supervised Learning
Convolutional Neural Networks
Primarily used for image analysis.
Inspired by animal visual cortex
Neurons respond to input from near neurons in what is called the
receptive field.

26. Deep Learning for Supervised Learning
Design of a Convolutional Neural Network CNN
A CNN has input, output and hidden units.
Hidden units can be of 3 types:
Convolutional
Pooling
Fully Connected
output
Convolutional
hidden Pooling
Fully Connected
input

27. Deep Learning for Supervised Learning
Design of a Convolutional Neural Network CNN
Why are CNNs important when dealing with images?
Assume an image of size 500x500 pixels and each pixel can have 3 color channels.
500 pixels
1 pixel w/ 3 color channels
500 pixels
A fully connected NN needs a hidden layer with 500x500x3 weights = 750,000 weights!

28. Deep Learning for Supervised Learning
Design of a Convolutional Neural Network CNN
Why are CNNs important when dealing with images?
Instead a CNN with have local set of weights only.
Each neuron will be connected to a few close by neurons only (idea of receptive field)

29. Deep Learning for Supervised Learning
Design of a Convolutional Neural Network CNN
First difference with traditional NN is that neurons are arranged in 3 dimensions.

30. Deep Learning for Supervised Learning
Design of a Convolutional Neural Network CNN

31. Deep Learning for Supervised Learning
Convolutional Neural Networks
Local Weight Update
Implies a sparse representation

32. Deep Learning for Supervised Learning
Convolutional Neural Networks
Convolution Operation
We need to learn the kernel K and share those parameters across the entire image.

33. Deep Learning for Supervised Learning
Convolutional Neural Networks
Vertical and horizontal filters:

34. Deep Learning for Supervised Learning
Convolutional Neural Networks

35. Deep Learning for Supervised Learning
Convolutional Neural Networks
Convolution Operation
We can learn rotated objects using convolution.

36. Deep Learning for Supervised Learning
Design of a Convolutional Neural Network CNN
Layers alternate between convolutional layers and pooling layers:

37. Deep Learning for Supervised Learning
Design of a Convolutional Neural Network CNN
Pooling aggressively reduces the dimensionality of the feature space.
The idea is as follows:
We partition the image into a set of non-overlapping rectangles.
For each region we simply output the maximum value of that region (set of pixels). This is called “Max Pooling”.

38. Deep Learning for Supervised Learning
Design of a Convolutional Neural Network CNN
Full convolutional neural network :
Apply convolution, pooling (or subsampling) iteratively.
Finally apply fully connected neural network:

39. Deep learning Introduction Classes of Deep Learning Networks
Unsupervised or Generative Learning
Supervised Learning
Hybrid Networks
References

40. Deep Networks for Hybrid Learning
Pre-trained Deep Neural Network

41. Deep learning Introduction Classes of Deep Learning Networks
Unsupervised or Generative Learning
Supervised Learning
Hybrid Networks
References

42. References [1] A Fast Learning Algorithm for Deep Belief Nets
by G. Hinton, S. Osindero, and Y. W. The.
IEEE Transactions on Neural Computation Vol. 18 pp , 2006.
[2] Representation Learning: A Review and New Perspectives
by Y. Bengio, A. Courville, and P. Vincent.
IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI),
Vol. 35 No. 8, pp – 1828, 2013.
[3] Deep Learning: Methods and Applications
Li Deng and Dong Yu, NOW PUBLISHERS, 2014
[4] Binary Coding of Speech Spectrograms Using a Deep Autoencoder.
L. Deng, M. Seltzer, D. Yu, A. Acer1, A. Mohamed, and G. Hinton. Interspeech, 2010.

43. References
[5] Deep Learning Neural Networks: Design and Case Studies by Daniel Graupe, World Scientific Publishing 2016.
[6] Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville, MIT Press, 2016.
[7] Deep Learning: A Practitioner’s Approach by Josh Patterson and Adam Gibson, O’Reilly Media Inc., 2017.
[8] Fundamentals of Deep Learning by Nikhil Buduma, O’Reilly Media Inc., 2017.