Presentation is loading. Please wait.

Presentation is loading. Please wait.

Matt Gormley Lecture 16 October 24, 2016

Published byBrooke Copeland Modified over 6 years ago

Similar presentations

Presentation on theme: "Matt Gormley Lecture 16 October 24, 2016"— Presentation transcript:

1 Matt Gormley Lecture 16 October 24, 2016
School of Computer Science 10-601B Introduction to Machine Learning Deep Learning (Part I) Matt Gormley Lecture 16 October 24, 2016 Readings: Nielsen (online book) Neural Networks and Deep Learning

2 Reminders Midsemester grades released today

3 Outline Deep Neural Networks (DNNs)
Three ideas for training a DNN Experiments: MNIST digit classification Autoencoders Pretraining Convolutional Neural Networks (CNNs) Convolutional layers Pooling layers Image recognition Recurrent Neural Networks (RNNs) Bidirectional RNNs Deep Bidirectional RNNs Deep Bidirectional LSTMs Connection to forward-backward algorithm Part I Part II

4 Pre-Training for Deep Nets

5 A Recipe for Machine Learning
Background A Recipe for Machine Learning Goals for Today’s Lecture 1. Given training data: 3. Define goal: Explore a new class of decision functions (Deep Neural Networks) Consider variants of this recipe for training 2. Choose each of these: Decision function Loss function 4. Train with SGD: (take small steps opposite the gradient)

6 Idea #1: No pre-training
Idea #1: (Just like a shallow network) Compute the supervised gradient by backpropagation. Take small steps in the direction of the gradient (SGD)

7 Comparison on MNIST Training
Results from Bengio et al. (2006) on MNIST digit classification task Percent error (lower is better)

8 Comparison on MNIST Training
Results from Bengio et al. (2006) on MNIST digit classification task Percent error (lower is better)

9 Idea #1: No pre-training
Idea #1: (Just like a shallow network) Compute the supervised gradient by backpropagation. Take small steps in the direction of the gradient (SGD) What goes wrong? Gets stuck in local optima Nonconvex objective Usually start at a random (bad) point in parameter space Gradient is progressively getting more dilute “Vanishing gradients”

10 Problem A: Nonconvexity
Training Problem A: Nonconvexity Where does the nonconvexity come from? Even a simple quadratic z = xy objective is nonconvex: z x y

11 Problem A: Nonconvexity
Training Problem A: Nonconvexity Where does the nonconvexity come from? Even a simple quadratic z = xy objective is nonconvex: z x y

12 Problem A: Nonconvexity
Training Problem A: Nonconvexity Stochastic Gradient Descent… …climbs to the top of the nearest hill…

13 Problem A: Nonconvexity
Training Problem A: Nonconvexity Stochastic Gradient Descent… …climbs to the top of the nearest hill…

14 Problem A: Nonconvexity
Training Problem A: Nonconvexity Stochastic Gradient Descent… …climbs to the top of the nearest hill…

15 Problem A: Nonconvexity
Training Problem A: Nonconvexity Stochastic Gradient Descent… …climbs to the top of the nearest hill…

16 Problem A: Nonconvexity
Training Problem A: Nonconvexity Stochastic Gradient Descent… …climbs to the top of the nearest hill… …which might not lead to the top of the mountain TODO: Important note: this surface is a function of the parameters – before we discussed surfaces which were functions of the input.

17 Problem B: Vanishing Gradients
Training Problem B: Vanishing Gradients The gradient for an edge at the base of the network depends on the gradients of many edges above it The chain rule multiplies many of these partial derivatives together y Output c1 c2 … cF Hidden Layer b1 b2 … bE Hidden Layer TODO: Drop this. Too much detail. TODO: Show vanishing gradients as less dark over the layers. Suppose WLOG we scale everything such that the gradients are less than one. What happens with backprop? TODO: Mention the activation function. Relu can already start to solve this problem. Let’s try to solve it another way though. a1 a2 aD … Hidden Layer x1 x2 x3 … xM Input

18 Problem B: Vanishing Gradients
Training Problem B: Vanishing Gradients The gradient for an edge at the base of the network depends on the gradients of many edges above it The chain rule multiplies many of these partial derivatives together y Output c1 c2 … cF Hidden Layer b1 b2 … bE Hidden Layer TODO: Drop this. Too much detail. TODO: Show vanishing gradients as less dark over the layers. Suppose WLOG we scale everything such that the gradients are less than one. What happens with backprop? TODO: Mention the activation function. Relu can already start to solve this problem. Let’s try to solve it another way though. a1 a2 aD … Hidden Layer x1 x2 x3 … xM Input

19 Problem B: Vanishing Gradients
Training Problem B: Vanishing Gradients The gradient for an edge at the base of the network depends on the gradients of many edges above it The chain rule multiplies many of these partial derivatives together y Output 0.1 c1 c2 … cF Hidden Layer 0.3 b1 b2 … bE Hidden Layer 0.2 TODO: Drop this. Too much detail. TODO: Show vanishing gradients as less dark over the layers. Suppose WLOG we scale everything such that the gradients are less than one. What happens with backprop? TODO: Mention the activation function. Relu can already start to solve this problem. Let’s try to solve it another way though. a1 a2 aD … Hidden Layer 0.7 x1 x2 x3 … xM Input

20 Idea #1: No pre-training
Idea #1: (Just like a shallow network) Compute the supervised gradient by backpropagation. Take small steps in the direction of the gradient (SGD) What goes wrong? Gets stuck in local optima Nonconvex objective Usually start at a random (bad) point in parameter space Gradient is progressively getting more dilute “Vanishing gradients”

21 Idea #2: Supervised Pre-training
Idea #2: (Two Steps) Train each level of the model in a greedy way Then use our original idea Supervised Pre-training Use labeled data Work bottom-up Train hidden layer 1. Then fix its parameters. Train hidden layer 2. Then fix its parameters. … Train hidden layer n. Then fix its parameters. Supervised Fine-tuning Use labeled data to train following “Idea #1” Refine the features by backpropagation so that they become tuned to the end-task TODO: doesn’t work  works a little better ; or won’t work when labeled data is limited.

22 Idea #2: Supervised Pre-training
Idea #2: (Two Steps) Train each level of the model in a greedy way Then use our original idea x1 a1 x2 x3 xM y a2 aD … Output Input Hidden Layer 1

23 Idea #2: Supervised Pre-training
Idea #2: (Two Steps) Train each level of the model in a greedy way Then use our original idea y Output b1 b2 bE … Hidden Layer 2 a1 a2 … aD Hidden Layer 1 x1 x2 x3 … xM Input

24 Idea #2: Supervised Pre-training
Idea #2: (Two Steps) Train each level of the model in a greedy way Then use our original idea y Output c1 c2 … cF Hidden Layer 3 b1 b2 bE … Hidden Layer 2 a1 a2 … aD Hidden Layer 1 x1 x2 x3 … xM Input

25 Idea #2: Supervised Pre-training
Idea #2: (Two Steps) Train each level of the model in a greedy way Then use our original idea y Output c1 c2 … cF Hidden Layer 3 b1 b2 bE … Hidden Layer 2 a1 a2 … aD Hidden Layer 1 x1 x2 x3 … xM Input

26 Comparison on MNIST Training
Results from Bengio et al. (2006) on MNIST digit classification task Percent error (lower is better)

27 Comparison on MNIST Training
Results from Bengio et al. (2006) on MNIST digit classification task Percent error (lower is better)

28 Idea #3: Unsupervised Pre-training
Idea #3: (Two Steps) Use our original idea, but pick a better starting point Train each level of the model in a greedy way Unsupervised Pre-training Use unlabeled data Work bottom-up Train hidden layer 1. Then fix its parameters. Train hidden layer 2. Then fix its parameters. … Train hidden layer n. Then fix its parameters. Supervised Fine-tuning Use labeled data to train following “Idea #1” Refine the features by backpropagation so that they become tuned to the end-task

29 The solution: Unsupervised pre-training
Unsupervised pre-training of the first layer: What should it predict? What else do we observe? The input! This topology defines an Auto-encoder. y Output a1 a2 aD … Hidden Layer x1 x2 x3 … xM Input

30 The solution: Unsupervised pre-training
Unsupervised pre-training of the first layer: What should it predict? What else do we observe? The input! This topology defines an Auto-encoder. x1 ’ x2 x3 … xM “Input” a1 a2 aD … Hidden Layer x1 x2 x3 … xM Input

31 Auto-Encoders Key idea: Encourage z to give small reconstruction error: x’ is the reconstruction of x Loss = || x – DECODER(ENCODER(x)) ||2 Train with the same backpropagation algorithm for 2-layer Neural Networks with xm as both input and output. x1 ’ x2 x3 … xM “Input” DECODER: x’ = h(W’z) a1 a2 aD … Hidden Layer ENCODER: z = h(Wx) x1 x2 x3 … xM Input Slide adapted from Raman Arora

32 The solution: Unsupervised pre-training
Work bottom-up Train hidden layer 1. Then fix its parameters. Train hidden layer 2. Then fix its parameters. … Train hidden layer n. Then fix its parameters. x1 ’ x2 x3 … xM “Input” a1 a2 aD … Hidden Layer x1 x2 x3 … xM Input

33 The solution: Unsupervised pre-training
Work bottom-up Train hidden layer 1. Then fix its parameters. Train hidden layer 2. Then fix its parameters. … Train hidden layer n. Then fix its parameters. a1 a2 aF … ’ b1 b2 … bF Hidden Layer a1 a2 aD … Hidden Layer x1 x2 x3 … xM Input

34 The solution: Unsupervised pre-training
b1 b2 bF … ’ Unsupervised pre-training Work bottom-up Train hidden layer 1. Then fix its parameters. Train hidden layer 2. Then fix its parameters. … Train hidden layer n. Then fix its parameters. c1 c2 cF … Hidden Layer b1 b2 … bF Hidden Layer a1 a2 aD … Hidden Layer x1 x2 x3 … xM Input

35 The solution: Unsupervised pre-training
Work bottom-up Train hidden layer 1. Then fix its parameters. Train hidden layer 2. Then fix its parameters. … Train hidden layer n. Then fix its parameters. Supervised fine-tuning Backprop and update all parameters y Output c1 c2 cF … Hidden Layer b1 b2 … bF Hidden Layer a1 a2 aD … Hidden Layer x1 x2 x3 … xM Input

36 Deep Network Training Idea #1: Idea #2: Idea #3:
Supervised fine-tuning only Idea #2: Supervised layer-wise pre-training Supervised fine-tuning Idea #3: Unsupervised layer-wise pre-training Supervised fine-tuning

37 Comparison on MNIST Training
Results from Bengio et al. (2006) on MNIST digit classification task Percent error (lower is better)

38 Comparison on MNIST Training
Results from Bengio et al. (2006) on MNIST digit classification task Percent error (lower is better)

39 Is layer-wise pre-training always necessary?
In 2010, a record on a hand-writing recognition task was set by standard supervised backpropagation (our Idea #1). How? A very fast implementation on GPUs. See Ciresen et al. (2010)

40 Deep Learning Goal: learn features at different levels of abstraction
Training can be tricky due to… Nonconvexity Vanishing gradients Unsupervised layer-wise pre-training can help with both!

Download ppt "Matt Gormley Lecture 16 October 24, 2016"

Similar presentations

Greedy Layer-Wise Training of Deep Networks

Greedy Layer-Wise Training of Deep Networks
Neural networks Introduction Fitting neural networks

Neural networks Introduction Fitting neural networks
Advanced topics.

Advanced topics.
Lecture 14 – Neural Networks

Lecture 14 – Neural Networks
How to do backpropagation in a brain

How to do backpropagation in a brain
Branch Prediction with Neural- Networks: Hidden Layers and Recurrent Connections Andrew Smith CSE Dept. June 10, 2004.

Branch Prediction with Neural- Networks: Hidden Layers and Recurrent Connections Andrew Smith CSE Dept. June 10, 2004.
CSC321: Introduction to Neural Networks and Machine Learning Lecture 20 Learning features one layer at a time Geoffrey Hinton.

CSC321: Introduction to Neural Networks and Machine Learning Lecture 20 Learning features one layer at a time Geoffrey Hinton.
How to do backpropagation in a brain

How to do backpropagation in a brain
A shallow introduction to Deep Learning

A shallow introduction to Deep Learning
Neural Network Introduction Hung-yi Lee. Review: Supervised Learning Training: Pick the “best” Function f * Training Data Model Testing: Hypothesis Function.

Neural Network Introduction Hung-yi Lee. Review: Supervised Learning Training: Pick the “best” Function f * Training Data Model Testing: Hypothesis Function.
Building high-level features using large-scale unsupervised learning Anh Nguyen, Bay-yuan Hsu CS290D – Data Mining (Spring 2014) University of California,

Building high-level features using large-scale unsupervised learning Anh Nguyen, Bay-yuan Hsu CS290D – Data Mining (Spring 2014) University of California,
Introduction to Deep Learning

Introduction to Deep Learning
Object Recognizing. Deep Learning Success in 2012 DeepNet and speech processing.

Object Recognizing. Deep Learning Success in 2012 DeepNet and speech processing.
Convolutional Neural Network

Convolutional Neural Network
CSC321 Lecture 24 Using Boltzmann machines to initialize backpropagation Geoffrey Hinton.

CSC321 Lecture 24 Using Boltzmann machines to initialize backpropagation Geoffrey Hinton.
CSC321 Lecture 27 Using Boltzmann machines to initialize backpropagation Geoffrey Hinton.

CSC321 Lecture 27 Using Boltzmann machines to initialize backpropagation Geoffrey Hinton.
Bump Hunting The objective PRIM algorithm Beam search References: Feelders, A.J. (2002). Rule induction by bump hunting. In J. Meij (Ed.), Dealing with.

Bump Hunting The objective PRIM algorithm Beam search References: Feelders, A.J. (2002). Rule induction by bump hunting. In J. Meij (Ed.), Dealing with.
Neural networks (2) Reminder Avoiding overfitting Deep neural network Brief summary of supervised learning methods.

Neural networks (2) Reminder Avoiding overfitting Deep neural network Brief summary of supervised learning methods.
Deep Learning Overview Sources:https://deeplearningworkshopnips2010.files.wordpress.com/2010/09/nips10- workshop-tutorial-final.pdf

Deep Learning Overview Sources: workshop-tutorial-final.pdf
Xintao Wu University of Arkansas Introduction to Deep Learning 1.

Xintao Wu University of Arkansas Introduction to Deep Learning 1.

Similar presentations

Ads by Google