1. Lecture 2 Introduction to Neural Networks and Fuzzy Logic President UniversityErwin SitompulNNFL 2/1 Dr.-Ing. Erwin Sitompul President University http://zitompul.wordpress.com 2014

2. President UniversityErwin SitompulNNFL 2/2 Weights, need to be determined Biological neuron Artificial neuron Bias, need to be determined Learning ProcessesNeural Networks Biological and Artificial Neuron

3. President UniversityErwin SitompulNNFL 2/3 Learning ProcessesNeural Networks Application of Neural Networks Function approximation and prediction Pattern recognition Signal processing Modeling and control Machine learning

4. President UniversityErwin SitompulNNFL 2/4 Building a Neural Network Select Structure: design the way that the neurons are interconnected. Select weights: decide the strengths with which the neurons are interconnected. Weights are selected to get a “good match” of network output to the output of a training set. Training set is a set of inputs and desired outputs. The weight selection is conducted by the use of a learning algorithm. Learning ProcessesNeural Networks

5. President UniversityErwin SitompulNNFL 2/5 Stage 1: Network Training Training Data Stage 2: Network Validation Artificial neural network Input and output sets, adequate coverage Learning Process In the form of a set of optimized synaptic weights and biases Unseen Data From the same range as the training data Artificial neural network Implementation Phase Learning ProcessesNeural Networks Learning Process Knowledge Output Prediction

6. President UniversityErwin SitompulNNFL 2/6 Learning Process Learning is a process by which the free parameters of a neural network are adapted through a process of stimulation by the environment in which the network is embedded. In most cases, due to complex optimization plane, the optimized weights and biases are obtained as a result of a number of learning iterations. [w,b][w,b] x y [w,b] 0 x y(0) Initialize: Iteration (0) [w,b] 1 x y(1) Iteration (1) [w,b] n x y(n) ≈ d Iteration (n) ANN d : desired output … Learning ProcessesNeural Networks

7. President UniversityErwin SitompulNNFL 2/7 Learning Rules Learning ProcessesNeural Networks Error Correction Learning Delta Rule or Widrow-Hoff Rule Memory Based Learning Nearest Neighbor Rule Hebbian Learning Synchronous activation increases the synaptic strength Asynchronous activation decreases the synaptic strength Competitive Learning Boltzmann Learning

8. President UniversityErwin SitompulNNFL 2/8 wk1(n)wk1(n) x1x1 x2x2 xmxm Inputs Synaptic weights Bias Activation function wk2(n)wk2(n) w km (n)  Output y k (n) Desired output d k (n) ek (n)ek (n)  f (.) bk(n)bk(n) 1  Error signal Learning ProcessesNeural Networks Error-Correction Learning  Learning Rule

9. President UniversityErwin SitompulNNFL 2/9 Learning ProcessesNeural Networks Delta Rule (Widrow-Hoff Rule) Minimization of a cost function (or performance index)

10. President UniversityErwin SitompulNNFL 2/10 w kj (0) = 0 y k (n) = [w kj (n) x j (n)] w kj (n+1) = w kj (n) + [d k (n) – y k (n)] x j (n)  : learning rate, [0…1] n = n+1 n = 0 Least Means Squares Rule Learning ProcessesNeural Networks Delta Rule (Widrow-Hoff Rule)

11. President UniversityErwin SitompulNNFL 2/11 Learning ProcessesNeural Networks Learning Paradigm  ANN Error Desired Actual   Environment (Data) Teacher (Expert) Supervised Unsupervised Environment (Data) Delay ANN Delayed Reinforcement Learning Cost Function

12. President UniversityErwin SitompulNNFL 2/12 Single Layer PerceptronsNeural Networks Single Layer Perceptrons Output unit is independent of the others. Analysis can be limited to single output perceptron. Single-layer perceptron network is a network with all the inputs connected directly to the output(s).

13. President UniversityErwin SitompulNNFL 2/13 Single Layer PerceptronsNeural Networks Derivation of a Learning Rule for Perceptrons w1w1 w2w2 E(w)E(w) Key idea: Learning is performed by adjusting the weights in order to minimize the sum of squared errors on a training. Weights are updated repeatedly (in each epoch/iteration). Sum of squared errors is a classical error measure (e.g. commonly used in linear regression). Learning can be viewed as an optimization search problem in weight space.

14. President UniversityErwin SitompulNNFL 2/14 Single Layer PerceptronsNeural Networks Derivation of a Learning Rule for Perceptrons The learning rule performs a search within the solution's vector space towards a global minimum.  The error surface itself is a hyper-paraboloid but is seldom as smooth as is depicted below.  In most problems, the solution space is quite irregular with numerous pits and hills which may cause the network to settle down in a local minimum (not the best overall solution).  Epochs are repeated until stopping criterion is reached (error magnitude, number of iterations, change of weights, etc).

15. President UniversityErwin SitompulNNFL 2/15 Single Layer PerceptronsNeural Networks Derivation of a Learning Rule for Perceptrons Widrow [1962] x1x1 x2x2 xmxm wk1wk1 wk2wk2 w km......  Adaline (Adaptive Linear Element) Goal:

16. President UniversityErwin SitompulNNFL 2/16 Least Mean Squares (LMS) Single Layer PerceptronsNeural Networks The following cost function (error function) should be minimized:

17. President UniversityErwin SitompulNNFL 2/17 Single Layer PerceptronsNeural Networks Least Mean Squares (LMS) Letting f(w k ) = f (w k1, w k2, …, w km ) be a function over R m, then Defining

18. President UniversityErwin SitompulNNFL 2/18 ff ww ff ww df : positive df : zero df : negative go uphill plain go downhill ff ww To minimize f, we choose Single Layer PerceptronsNeural Networks Gradient Operator df is thus guaranteed to be always negative

19. President UniversityErwin SitompulNNFL 2/19 Single Layer PerceptronsNeural Networks Adaline Learning Rule With then As already obtained before, Weight Modification Rule Defining we can write

20. President UniversityErwin SitompulNNFL 2/20 Single Layer PerceptronsNeural Networks Adaline Learning Modes Batch Learning Mode Incremental Learning Mode

21. President UniversityErwin SitompulNNFL 2/21   - Learning Rule  LMS Algorithm  Widrow-Hoff Learning Rule Single Layer PerceptronsNeural Networks Adaline Learning Rule

22. President UniversityErwin SitompulNNFL 2/22 Single Layer PerceptronsNeural Networks Generalization and Early Stopping By proper training, a neural network may produce reasonable output for inputs not seen during training  Generalization Generalization is particularly useful for the analysis of a “noisy” data (e.g. time–series ) “Overtraining” will not improve the ability of a neural network to produce good output. On the contrary, it will try to take noise as the real data and lost its generality.

23. President UniversityErwin SitompulNNFL 2/23 Generalization and Early Stopping Single Layer PerceptronsNeural Networks Overfitting vs Generalization

24. President UniversityErwin SitompulNNFL 2/24 Homework 2 Single Layer PerceptronsNeural Networks Given a function y = 4x 2, you are required to find the value of x that will result y = 2 by using the Least Mean Squares method. Use initial estimate x 0 = 1 and learning rate η = 0.01. Write down the results of the first 10 epochs/iterations. Give conclusion about your result. Note: Calculation can be done manually or using Matlab.

25. President UniversityErwin SitompulNNFL 2/25 Homework 2A Single Layer PerceptronsNeural Networks Given a function y = 2x 3 + cos 2 x, you are required to find the value of x that will result y = 5 by using the Least Mean Squares method. Use initial estimate x 0 = 0.2*Student ID and learning rate η = 0.01. Write down the results of the first 10 epochs/iterations. Give conclusion about your result. Note: Calculation can be done manually or using Matlab/Excel.