Presentation is loading. Please wait.

Presentation is loading. Please wait.

Gradient Checks for ANN

Similar presentations


Presentation on theme: "Gradient Checks for ANN"β€” Presentation transcript:

1 Gradient Checks for ANN
Yujia Bao Mar 7, 2017

2 Finite Difference Let 𝑓(π‘₯) be any differentiable function, we can approximate its derivative by d𝑓(π‘₯) dπ‘₯ = 𝑓 π‘₯+πœ– βˆ’π‘“(π‘₯βˆ’πœ–) 2πœ– +𝑂( πœ– 2 ) for some very small number πœ–.

3 How to compare the numerical gradient with the analytic gradient?

4 Relative Error Let 𝑓 𝑛 β€² be the numerical gradient calculated using finite difference, and 𝑓 π‘Ž β€² be the analytic gradient calculated using back prop. Define the relative error π‘’π‘Ÿπ‘Ÿπ‘œπ‘Ÿ= 𝑓 π‘Ž β€² βˆ’ 𝑓 𝑛 β€² max 𝑓 π‘Ž β€² , 𝑓 𝑛 β€² , π‘’π‘π‘ π‘–π‘™π‘œπ‘›

5 Relative Error π‘’π‘Ÿπ‘Ÿπ‘œπ‘Ÿ> 10 βˆ’4 usually means the analytic gradient is wrong. π‘’π‘Ÿπ‘Ÿπ‘œπ‘Ÿ< 10 βˆ’4 is fine for sigmoid activation (including logistic, tanh, softmax). But if you are using (leaky) ReLU, then 10 βˆ’4 might be too large. π‘’π‘Ÿπ‘Ÿπ‘œπ‘Ÿ< 10 βˆ’7 means your analytic gradient is correct.

6 Debugging Procedure Goal: Check the gradient for a single weight 𝑀 is computed correctly. Given: One example (Input features with a label) Forward prop and Backward prop to get the gradient for 𝑀. Let 𝑀←𝑀+πœ– (I usually choose πœ–= 10 βˆ’5 ). Forward prop to get the output, and then compute the loss. Let π‘€β†π‘€βˆ’2πœ– (Now 𝑀 is 𝑀 π‘œπ‘Ÿπ‘–π‘”π‘–π‘› βˆ’πœ–). Check the relative error. Recover the origin weight by 𝑀←𝑀+πœ–.

7 Debugging Procedure Suppose our network has the following structure:
Input -> Conv1 -> Pool1 -> Conv2 -> Pool2 -> ReLU -> Output If the gradients from Input to Conv1 are correct, then we are done! Otherwise, we check the gradients from Pool1 to Conv2 (since there is no weights from Conv1 to Pool1). If it is correct, then this means there are some bugs in our Back prop code from Pool1 to Input. And so on…

8 Thanks.


Download ppt "Gradient Checks for ANN"

Similar presentations


Ads by Google