-Artificial Neural Network- Chapter 3 Perceptron 朝陽科技大學 資訊管理系 李麗華 教授.

Slides:



Advertisements
Similar presentations
Multi-Layer Perceptron (MLP)
Advertisements

Perceptron Lecture 4.
Slides from: Doug Gray, David Poole
NEURAL NETWORKS Backpropagation Algorithm
1 Machine Learning: Lecture 4 Artificial Neural Networks (Based on Chapter 4 of Mitchell T.., Machine Learning, 1997)
Artificial Neural Networks (1)
Perceptron Learning Rule
1 Neural networks. Neural networks are made up of many artificial neurons. Each input into the neuron has its own weight associated with it illustrated.
Multilayer Perceptrons 1. Overview  Recap of neural network theory  The multi-layered perceptron  Back-propagation  Introduction to training  Uses.
-Artificial Neural Network- Chapter 2 Basic Model
Artificial Neural Networks
-Artificial Neural Network- Counter Propagation Network
Simple Neural Nets For Pattern Classification
Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley.
The back-propagation training algorithm
S. Mandayam/ ANN/ECE Dept./Rowan University Artificial Neural Networks ECE /ECE Fall 2008 Shreekanth Mandayam ECE Department Rowan University.
20.5 Nerual Networks Thanks: Professors Frank Hoffmann and Jiawei Han, and Russell and Norvig.
September 30, 2010Neural Networks Lecture 8: Backpropagation Learning 1 Sigmoidal Neurons In backpropagation networks, we typically choose  = 1 and 
-Artificial Neural Network- Adaptive Resonance Theory(ART) 朝陽科技大學 資訊管理系 李麗華 教授.
Chapter 6: Multilayer Neural Networks
-Artificial Neural Network- Chapter 4 朝陽科技大學 資訊管理系 李麗華 教授.
Before we start ADALINE
Data Mining with Neural Networks (HK: Chapter 7.5)
-Artificial Neural Network- Chapter 5 Back Propagation Network
CHAPTER 11 Back-Propagation Ming-Feng Yeh.
CS 4700: Foundations of Artificial Intelligence
Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.
-Artificial Neural Network- Chapter 3 Perceptron 朝陽科技大學 資訊管理系 李麗華教授.
CS623: Introduction to Computing with Neural Nets (lecture-10) Pushpak Bhattacharyya Computer Science and Engineering Department IIT Bombay.
Neural Networks. Plan Perceptron  Linear discriminant Associative memories  Hopfield networks  Chaotic networks Multilayer perceptron  Backpropagation.
Artificial Neural Networks
-Artificial Neural Network- Chapter 9 Self Organization Map(SOM) 朝陽科技大學 資訊管理系 李麗華 教授.
-Artificial Neural Network- Hopfield Neural Network(HNN) 朝陽科技大學 資訊管理系 李麗華 教授.
1 Chapter 6: Artificial Neural Networks Part 2 of 3 (Sections 6.4 – 6.6) Asst. Prof. Dr. Sukanya Pongsuparb Dr. Srisupa Palakvangsa Na Ayudhya Dr. Benjarath.
Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley.
1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology.
 Diagram of a Neuron  The Simple Perceptron  Multilayer Neural Network  What is Hidden Layer?  Why do we Need a Hidden Layer?  How do Multilayer.
ECE 8443 – Pattern Recognition ECE 8527 – Introduction to Machine Learning and Pattern Recognition LECTURE 16: NEURAL NETWORKS Objectives: Feedforward.
Artificial Intelligence Techniques Multilayer Perceptrons.
1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology.
Artificial Neural Networks. The Brain How do brains work? How do human brains differ from that of other animals? Can we base models of artificial intelligence.
A note about gradient descent: Consider the function f(x)=(x-x 0 ) 2 Its derivative is: By gradient descent. x0x0 + -
Introduction to Artificial Intelligence (G51IAI) Dr Rong Qu Neural Networks.
CS344: Introduction to Artificial Intelligence (associated lab: CS386) Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 31: Feedforward N/W; sigmoid.
Feed-Forward Neural Networks 主講人 : 虞台文. Content Introduction Single-Layer Perceptron Networks Learning Rules for Single-Layer Perceptron Networks – Perceptron.
CS344: Introduction to Artificial Intelligence (associated lab: CS386) Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 30: Perceptron training convergence;
Back-Propagation Algorithm AN INTRODUCTION TO LEARNING INTERNAL REPRESENTATIONS BY ERROR PROPAGATION Presented by: Kunal Parmar UHID:
CS621 : Artificial Intelligence
Chapter 2 Single Layer Feedforward Networks
Introduction to Neural Networks Introduction to Neural Networks Applied to OCR and Speech Recognition An actual neuron A crude model of a neuron Computational.
Artificial Neural Networks Chapter 4 Perceptron Gradient Descent Multilayer Networks Backpropagation Algorithm 1.
EEE502 Pattern Recognition
IE 585 History of Neural Networks & Introduction to Simple Learning Rules.
1 Perceptron as one Type of Linear Discriminants IntroductionIntroduction Design of Primitive UnitsDesign of Primitive Units PerceptronsPerceptrons.
Perceptrons Michael J. Watts
Previous Lecture Perceptron W  t+1  W  t  t  d(t) - sign (w(t)  x)] x Adaline W  t+1  W  t  t  d(t) - f(w(t)  x)] f’ x Gradient.
Artificial Intelligence CIS 342 The College of Saint Rose David Goldschmidt, Ph.D.
CS623: Introduction to Computing with Neural Nets (lecture-9) Pushpak Bhattacharyya Computer Science and Engineering Department IIT Bombay.
Pattern Recognition Lecture 20: Neural Networks 3 Dr. Richard Spillman Pacific Lutheran University.
CSE343/543 Machine Learning Mayank Vatsa Lecture slides are prepared using several teaching resources and no authorship is claimed for any slides.
Neural networks.
Fall 2004 Backpropagation CS478 - Machine Learning.
CS621: Artificial Intelligence
Prof. Carolina Ruiz Department of Computer Science
Data Mining with Neural Networks (HK: Chapter 7.5)
Neural Networks Chapter 5
-Artificial Neural Network- Perceptron (感知器)
Prof. Carolina Ruiz Department of Computer Science
Outline Announcement Neural networks Perceptrons - continued
Presentation transcript:

-Artificial Neural Network- Chapter 3 Perceptron 朝陽科技大學 資訊管理系 李麗華 教授

朝陽科技大學 李麗華 教授 2 Introduction of Perceptron In 1957, Rosenblatt and several other researchers developed perceptron, which used the similar network as proposed by McCulloch, and the learning rule for training network to solve pattern recognition problem. (*) But, this model was later criticized by Minsky who proved that it cannot solve the XOR problem.

朝陽科技大學 李麗華 教授 3 Perceptron Network Structure W 21 W 12 W 11 X1X1 X2X2 Y1Y1 f1f1 f2f2 f3f3 W 22 W 13 W 23

朝陽科技大學 李麗華 教授 4 Perceptron Training Process 1.Set up the network input nodes, layer(s), & connections 2.Randomly assign weights to W ij & bias: 3.Input training sets X i (preparing T j for verification ) 4.Training computation: 1 net j > 0 if Yj=Yj= 0 net j 0

朝陽科技大學 李麗華 教授 5 The training process 4.Training computation: If than: 5. repeat 3 ~5 until every input pattern is satisfied Update weights and bias :

朝陽科技大學 李麗華 教授 6 The recall process After the network has trained as mentioned above, any input vector X can be send into the perceptron network. The trained weights, W ij, and the bias, θ j, is used to derive net j and, therefore, the output Y j can be obtained for pattern recognition.

Exercise1: Solving the OR problem Let the training patterns are as follows. X1X1 X2X2 f X1X1 X2X2 T W 11 W 21 X2X2 X1X1 Y f

Let W 11 =1, W 21 =0.5, Θ =0.5 Let learning rate η=0.1 The initial net function is: net = W 11 X 1 + W 21 X 2 - Θ i.e. net = 1 X X Feed the input pattern into network one by one (0,0), net = -0.5, Y= 0, (T-Y) = 0 O.K. (0,1),net= 0, Y= 0,(T-Y) =1- Ø=1 Need to update weight (1,0),net= 0.5, Y=1,(T-Y) = 0 O.K. (1,1),net= 1, Y=1,(T-Y) = 0 O.K. update weights for pattern (0,1) ΔW 11 =(0.1)(1)( 0)= 0, W 11 = W 11+ ΔW 11= 1 ΔW 12 =(0,1)(1)( 1)= 0.1, W 21 = =0.6 ΔΘ=-(0.1)(1)=-0.1 Θ= =0.4

Applying new weights to the net function: net=1X X Verify the pattern (0,1) to see if it satisfies the expected output. (0,1),net= 0.2,Y= 1, (T-Y)=Ø O.K. Feed the next input pattern, again, one by one (1,0),net= 0.6,Y=1, (T-Y)= Ø O.K. (1,1),net= 1.2,Y=1, (T-Y)= Ø O.K. Since the first pattern(0,0) has not been testified with the new weights, feed again. (0,0), net=-0.4,Y=Ø,δ= ØO.K. Now all the patterns are satisfied. Hence, the network is successfully trained for OR patterns. We get the final network is : net=1X X (This is not the only solution, other solutions are possible.)

The trained network is formed as follow: Θ=0.4 X2X2 X1X1 1 0.6 Y Recall process: Once the network is trained, we can apply any two element vectors as a pattern and feed the pattern into the network for recognition. For example, we can feed (1,0) into to the network (1,0),net= 0.6, Y=1 Therefore, this pattern is recognized as 1. f

朝陽科技大學 李麗華 教授 11 Exercise2: Solving the AND problem Let the training patterns are used as follow X1X1 X2X2 T f1f1 X2X2 X1X1

Sol: Let W 11 =0.5, W 21 =0.5, Θ=1, Let η=0.1 The initial net function is: net =0.5X X 21 – 1 Feed the input pattern into network one by one (0,0), net=-1, Y=Ø, (T-Y)= Ø, O.K. (0,1),net=-0.5, Y= Ø, (T-Y)= Ø, O.K. (1,0),net=- 0.5, Y= Ø, (T-Y)= Ø, O.K. (1,1),net= Ø, Y= Ø, (T-Y)= 1, Need to update weight update weights for pattern (1,1) which does not satisfying the expected output: ΔW 11 =(0,1)(1)( 1)= 0.1, W 11 =0.6 ΔW 21 =(0,1)(1)( 1)= 0.1, W 21 = =0.6, ΔΘ=-(0.1)(1)=-0.1, Θ=1-0.1=0.9

Applying new weights to the net function: net=0.6X X Verify the pattern (1,1) to see if it satisfies the expected output. (1,1),net= 0.3, Y= 1,δ= ØO.K. Since the previous patterns are not testified with the new weights, feed them again. (0,0), net=-0.9 Y=Ø,δ= ØO.K. (0,1),net=-0.3 Y= Ø,δ=ØO.K. (1,0),net=- 0.3, Y= Ø,δ= ØO.K. We can generate the pattern recognition function for OR pattern is: net= 0.6X X Θ =0.9 X2X2 X1X1 0.6 Y

朝陽科技大學 李麗華 教授 14 Example: Solving the XOR problem Let the training patterns are used as follow. X1X1 X2X2 T P2P2 P4P4 P3P3 f2f2 P1P1 X2X2 X1X1 f1f1 f3f3 f1f1 P2, P3 P4 f2f2 P1

Let W 11 =1.0, W 21 = -1.0, Θ =0 If we choose one layer network, it will be proved that the network cannot be converged. This is because the XOR problem is a non-linear problem, i.e., one single linear function is not enough to recognize the pattern. Therefore, the solution is to add one hidden layer for extra functions. Θ3Θ3 f3f3 f1f1 f2f2 W 21 W 12 X2X2 X1X1 W 11 W 22 Θ1Θ1 Θ2Θ2

The following pattern is formed. So we solve f 1 as AND problem and f 2 as OR problem Assume we found the weights are: W 11 =0.3, W 21 =0.3, W 12 = 1, W 22 = 1 θ 1 =0.5 θ 2 =0.2 Therefore the network function for f1 & f2 is: f 1 = 0.3X X f 2 = 1 X X X1X1 X2X2 f1f1 f2f2 T3T

Now we need to feed the input one by one for training the network for f1 and f2 seprearately. This is to satisfiying the expected output for f1 using T1 and for f2 using T2. Finally, we use the output of f1 and f2 as input pattern to train the network for f 3., We can derive the result as following. f 3 =1 X X Θ =0.2 Θ =0.5 f1f Y f2f X2X2 X1X1 1 Θ = -0.1

Perceptron Learning of the Logical AND Function

Perceptron Learning of the Logical OR Function

Perceptron Attempt at Learning the Logical XOR Function  The one-layer network cannot solve the XOR problem…..the network goes to the infinite loop