1. Question about Gradient descent Hung-yi Lee

2. Larger gradient, larger steps? Best step:

3. Contradiction Original Gradient descent Adagrad RMSprop Larger gradient, larger step Divided by first derivative

4. Second Derivative Best step: The best step is |First derivative| Second derivative

5. More than one parameters |First derivative| Second derivative The best step is a b c d c < ac < a c > d Larger second derivative smaller second derivative a > b

6. What to do with Adagrad and RMSprop? |First derivative| Second derivative The best step is Use first derivative to estimate second derivative larger second derivative smaller second derivative

7. Acknowledgement This question is raised by 李廣和

8. Thanks for your attention!