Presentation is loading. Please wait.

Presentation is loading. Please wait.

NN & Optimization Yue Wu.

Published byYuliani Oesman Modified over 5 years ago

Similar presentations

Presentation on theme: "NN & Optimization Yue Wu."— Presentation transcript:

1 NN & Optimization Yue Wu

2 Questions?

3 Back Prop (Isn’t This Trivial?)
When I first understood what backpropagation was, my reaction was: “Oh, that’s just the chain rule! How did it take us so long to figure out?” I’m not the only one who’s had that reaction. It’s true that if you ask “is there a smart way to calculate derivatives in feedforward neural networks?” the answer isn’t that difficult. But I think it was much more difficult than it might seem. You see, at the time backpropagation was invented, people weren’t very focused on the feedforward neural networks that we study. It also wasn’t obvious that derivatives were the right way to train them. Those are only obvious once you realize you can quickly calculate derivatives. There was a circular dependency.

4 Gradient Descent

5 Backprop for Sigmoid

6

7 Computation Graphs Towards Auto-differentiation (Autograd)

8 Suppose we

9

10

11

12

13 Gradient Descent

14

15 Does it work?

16 Gradient!!!

17 Why Doesn’t Gradient Tell You Everything

18 Local Min & Saddle Points!

19 Saddle Point

20 Momentum

21 Hessian Matrix

22 Hessian Free Optimization Methods

23 Vanishing & Exploding GRADIENT!

24

25 Covariant Shift (BatchNorm)

26

27 GET MORE DATA!!!!

28 Dropout

Download ppt "NN & Optimization Yue Wu."

Similar presentations

Complex Functions These derivatives involve embedded composite, product, and quotient rules. The functions f or g must be derived using another rule.

Complex Functions These derivatives involve embedded composite, product, and quotient rules. The functions f or g must be derived using another rule.
NEURAL NETWORKS Backpropagation Algorithm

NEURAL NETWORKS Backpropagation Algorithm
Do now as a warm-up: Place an M&M in your mouth, but do NOT bite it. How is an M&M like a composite function? Write your answer on the index card. You.

Do now as a warm-up: Place an M&M in your mouth, but do NOT bite it. How is an M&M like a composite function? Write your answer on the index card. You.
Supervised learning 1.Early learning algorithms 2.First order gradient methods 3.Second order gradient methods.

Supervised learning 1.Early learning algorithms 2.First order gradient methods 3.Second order gradient methods.
September 30, 2010Neural Networks Lecture 8: Backpropagation Learning 1 Sigmoidal Neurons In backpropagation networks, we typically choose  = 1 and 

September 30, 2010Neural Networks Lecture 8: Backpropagation Learning 1 Sigmoidal Neurons In backpropagation networks, we typically choose  = 1 and 
Back-Propagation Algorithm

Back-Propagation Algorithm
Biointelligence Laboratory, Seoul National University

Biointelligence Laboratory, Seoul National University
Appendix B: An Example of Back-propagation algorithm

Appendix B: An Example of Back-propagation algorithm
Neural Network Introduction Hung-yi Lee. Review: Supervised Learning Training: Pick the “best” Function f * Training Data Model Testing: Hypothesis Function.

Neural Network Introduction Hung-yi Lee. Review: Supervised Learning Training: Pick the “best” Function f * Training Data Model Testing: Hypothesis Function.
CS 478 – Tools for Machine Learning and Data Mining Backpropagation.

CS 478 – Tools for Machine Learning and Data Mining Backpropagation.
Non-Bayes classifiers. Linear discriminants, neural networks.

Non-Bayes classifiers. Linear discriminants, neural networks.
11 1 Backpropagation. 11 2 Multilayer Perceptron R – S 1 – S 2 – S 3 Network.

11 1 Backpropagation Multilayer Perceptron R – S 1 – S 2 – S 3 Network.
Instructor: Prof. Pushpak Bhattacharyya 13/08/2004 CS-621/CS-449 Lecture Notes CS621/CS449 Artificial Intelligence Lecture Notes Set 4: 24/08/2004, 25/08/2004,

Instructor: Prof. Pushpak Bhattacharyya 13/08/2004 CS-621/CS-449 Lecture Notes CS621/CS449 Artificial Intelligence Lecture Notes Set 4: 24/08/2004, 25/08/2004,
Backpropagation Training

Backpropagation Training
Neural Networks Vladimir Pleskonjić 3188/2015. 2/20 Vladimir Pleskonjić General Feedforward neural networks Inputs are numeric features Outputs are in.

Neural Networks Vladimir Pleskonjić 3188/ /20 Vladimir Pleskonjić General Feedforward neural networks Inputs are numeric features Outputs are in.
Previous Lecture Perceptron W  t+1  W  t  t  d(t) - sign (w(t)  x)] x Adaline W  t+1  W  t  t  d(t) - f(w(t)  x)] f’ x Gradient.

Previous Lecture Perceptron W  t+1  W  t  t  d(t) - sign (w(t)  x)] x Adaline W  t+1  W  t  t  d(t) - f(w(t)  x)] f’ x Gradient.
Intro. ANN & Fuzzy Systems Lecture 11. MLP (III): Back-Propagation.

Intro. ANN & Fuzzy Systems Lecture 11. MLP (III): Back-Propagation.
Today’s Lecture Neural networks Training

Today’s Lecture Neural networks Training
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift CS838.

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift CS838.
Chapter 14 Partial Derivatives.

Chapter 14 Partial Derivatives.

Similar presentations

Ads by Google