1. Deep Learning Authors: Yann LeCun, Yoshua Bengio, Geoffrey Hinton
Presented by Nick Harris

2. Outline The Success of Deep Learning The Basics of Neural Networks
Forward and Back propagation
Supervised Learning
Convolutional Neural Networks
Recurrent Neural Networks
The Future of Deep Learning

3. Success of Deep Learning
Deep Learning has become increasingly popular in recent years
It is increasingly used in consumer products such as smartphones, and in commercial websites
It has also greatly improved state-of-the-art performance in a wide variety of tasks such as speech and object recognition
What makes Deep Learning so successful?

4. Success (continued)
Deep Learning offers numerous advantages over older conventional machine learning techniques:
Deep Learning techniques have a greater capacity to process raw data
Conventional techniques often require a lot of domain expertise and precise engineering for good pattern recognition. NNs on the other hand can perform well with little prior knowledge.
Deep Learning techniques tend to continue to improve when given vastly larger amounts of data
Two Deep Learning models that have been especially successful are the Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN)

5. Neural Networks
Neural Network systems consist of large networks of artificial neurons
These neurons map a set of inputs X to an output y
Parameters: weights (w), bias (b)
Activation Function (phi) to introduce nonlinearity

6. Neural Networks (continued)
Some popular activation functions:

7. Neural Networks (continued)
Many neurons are combined to form a network.
Maps an arbitrary number of inputs X to and arbitrary number of outputs Y
A Neural Network with even one hidden layer is sufficient to compute any function
Deep Learning is where many successive layers are used.
These many layers learn increasingly abstract representation of the data
Learn much better than small network in practice

8. Forward/Back Propagation
Forward Propagation is the process of activating the network with to produce an output
Back Propagation is the process of learning by adjusting the weights of the network

9. Gradient Descent
Learning can be thought of as finding an optimal point in a high dimensional landscape
Computing the error gradient and adjusting the weights can be thought of as “hill-climbing” where we move progressively toward the optimal point
Stochastic Gradient Descent is a technique used in practice that speeds up learning by computing the average gradient over a random set of samples (but introduces noise).

10. Supervised Learning
One of the most common learning processes for neural networks
Asks the network to make a prediction, then use the correct answer to adjust the weights with back propagation
Requires large data sets

11. Convolutional Neural Nets (CNNs)
A special class of neural networks that excel at processing images
Uses Convolutional and Pooling layers to break down images

12. CNNs (continued)
Convolutional Layer
Pooling Layer

13. Value of CNNs
CNNs have proven to be extremely good at image classification and object detection. As a result they are widely used in the computer vision field.

14. Recurrent Neural Networks (RNNs)
RNNs are a class of neural networks that excel at processing sequential data (sensitive to time and order)
One especially successful application is natural language processing
This includes predicting the next character of text, or the next word in a sequence
A more sophisticated application is translation by mapping text to a “thought vector”, then mapping to text in another language

15. RNNs (continued)
RNNs process input in a sequence one step at a time, receiving inputs from previous time steps
Maintain a hidden “state vector” that contains information about the history of past elements in the sequence
RNNs were initially problematic to train, but recent advances in architecture and training have allowed them to produce state-of-the-art results
Can be augmented with a more explicit memory to form an Long Short Term Memory (LSTM) network, which has proven to be more powerful

16. Future of Deep Learning
Unsupervised Learning is not as popular as Supervised of late, but is considered to have great potential
Interest in combining CNNs and RNNs with reinforcement learning for difficult computer vision tasks
Interest in improving natural language processing performance

17. Questions?