1. Neural NetworksNN 11 Neural Networks Teacher: Elena Marchiori R4.47 elena@cs.vu.nl Assistant: Kees Jong S2.22 cjong@cs.vu.nl

2. Neural NetworksNN 12 Course Outline Basics of neural network theory and practice for supervised and unsupervised learning. Most popular Neural Network models: architectures learning algorithms applications

3. Neural NetworksNN 13 Course Outline Rules:- 4 s.p - Final mark is based on two assignments, which will be available at the end of the course. - one assignment is on theory (to do alone). - one assignment is on practice (to do in couples). - Programming in Matlab 5.3. - Registration: send email to cjong@cs.vu.nl

4. Neural NetworksNN 14 Course Organization There is no text book. Course schedule, slides and exercises will be available at http://www.cs.vu.nl/~elena/nn.html

5. Neural NetworksNN 15 Neural Networks A NN is a machine learning approach inspired by the way in which the brain performs a particular learning task: –Knowledge about the learning task is given in the form of examples. –Inter neuron connection strengths (weights) are used to store the acquired information (the training examples). –During the learning process the weights are modified in order to model the particular learning task correctly on the training examples.

6. Neural NetworksNN 16 Supervised Learning –Recognizing hand-written digits, pattern recognition, regression. –Labeled examples (input, desired output) –Neural Network models: perceptron, feed-forward, radial basis function, support vector machine. Unsupervised Learning –Find similar groups of documents in the web, content addressable memory, clustering. –Unlabeled examples (different realizations of the input alone) –Neural Network models: self organizing maps, Hopfield networks. Learning

7. Neural NetworksNN 17 Network architectures Three different classes of network architectures –single-layer feed-forward neurons are organized –multi-layer feed-forward in acyclic layers –recurrent The architecture of a neural network is linked with the learning algorithm used to train

8. Neural NetworksNN 18 Single Layer Feed-forward Input layer of source nodes Output layer of neurons

9. Neural NetworksNN 19 Multi layer feed-forward Input layer Output layer Hidden Layer 3-4-2 Network

10. Neural NetworksNN 110 Recurrent Network with hidden neuron(s): unit delay operator z -1 implies dynamic system z -1 Recurrent network input hidden output

11. Neural NetworksNN 111 Neural Network Architectures

12. Neural NetworksNN 112 The Neuron The neuron is the basic information processing unit of a NN. It consists of: 1A set of synapses or connecting links, each link characterized by a weight: W 1, W 2, …, W m 2An adder function (linear combiner) which computes the weighted sum of the inputs: 3Activation function (squashing function) for limiting the amplitude of the output of the neuron.

13. Neural NetworksNN 113 The Neuron Input signal Synaptic weights Summing function Bias b Activation function Local Field v Output y x1x1 x2x2 xmxm w2w2 wmwm w1w1

14. Neural NetworksNN 114 Bias of a Neuron Bias b has the effect of applying an affine transformation to u v = u + b v is the induced field of the neuron v u

15. Neural NetworksNN 115 Bias as extra input Input signal Synaptic weights Summing function Activation function Local Field v Output y x1x1 x2x2 xmxm w2w2 wmwm w1w1 w0w0 x 0 = +1 Bias is an external parameter of the neuron. Can be modeled by adding an extra input.

16. Neural NetworksNN 116 Dimensions of a Neural Network Various types of neurons Various network architectures Various learning algorithms Various applications

17. Neural NetworksNN 117 Face Recognition 90% accurate learning head pose, and recognizing 1-of-20 faces

18. Neural NetworksNN 118 Handwritten digit recognition