1. Artificial Intelligence 10. Neural Networks
Japan Advanced Institute of Science and Technology (JAIST)
Yoshimasa Tsuruoka

2. Outline Regression Neural networks Lecture slides Linear regression
Gradient descent
Neural networks
Back propagation
Lecture slides

3. Linear regression Input: vector Output: numerical value Example
Predict the level of comfortableness from temperature and humidity
Temperature
Humidity
Comfortable 27
45%
25% 32
82% 3% 20
53%
78% 13
34%
18%

4. Optimizing the weight vector
Minimize the sum of squared errors

5. Gradient descent
Move in the direction of the negative gradient

6. Optimizing the weight vector
Squared errors summed over the whole training samples
Squared error on a particular sample n
Stochastic gradient computed from sample n

7. Neural networks Two-layer neural network Hidden Layer Output Input
Activation
Input
Output
Output

8. Activation function
Transforms the activation level of a unit into an output

9. Optimizing the weight vector
Error w.r.t. a particular sample n
Gradient
First layer
Second layer

10. Gradient
Second layer
Error

11. Gradient
First layer

12. Gradient
In summary,
Error in the first layer

13. Back propagation Backward propagation of errors
The same technique can be applied
to neural networks with more than
one layer of hidden units

14. Neural networks Capacity of approximating an arbitrary function
Prone to overfitting
The error function is not convex
Gradient descent can only give you local minima

15. Questionnaires
Lecture code　I2152