1. 1 Introduction to Artificial Neural Networks Andrew L. Nelson Visiting Research Faculty University of South Florida

2. 2/9/2004Neural Networks2 Overview Outline to the left Current topic in red Introduction History and Origins Biologically Inspired Applications Perceptron Activation Functions Hidden Layer Networks Training with BP Examples References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

3. 2/9/2004Neural Networks3 References W. S. McCulloch, W. Pitts, "A logical calculus of the ideas immanent in nervous activity," Bulletin of Mathematical Biophysics, vol. 5 pp. 115-133, 1943. J. L. McClelland, D. E. Rumelhart, Parallel Distributed Processing: Explorations in the Microstructure of Cognition, The MIT Press, 1986. C. M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995. References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

4. 2/9/2004Neural Networks4 Introduction Artificial Neural Networks (ANN) Connectionist computation Parallel distributed processing Computational models Biologically Inspired computational models Machine Learning Artificial intelligence References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

5. 2/9/2004Neural Networks5 History McCulloch and Pitts introduced the Perceptron in 1943. Simplified model of a biological neuron Fell out of favor in the late 1960's (Minsky and Papert) Perceptron limitations Resurgence in the mid 1980's Nonlinear Neuron Functions Back-propagation training References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

6. 2/9/2004Neural Networks6 Summary of Applications Function approximation Pattern recognition Signal processing Modeling Control Machine learning References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

7. 2/9/2004Neural Networks7 Biologically Inspired Electro-chemical signals Threshold output firing References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

8. 2/9/2004Neural Networks8 The Perceptron Binary classifier functions Threshold activation function References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

9. 2/9/2004Neural Networks9 The Perceptron: Threshold Activation Function Binary classifier functions Threshold activation function References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

10. 2/9/2004Neural Networks10 Linear Activation functions Output is scaled sum of inputs References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

11. 2/9/2004Neural Networks11 Nonlinear Activation Functions Sigmoid Neuron unit function References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

12. 2/9/2004Neural Networks12 Layered Networks. References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

13. 2/9/2004Neural Networks13 SISO Single Hidden Layer Network Can represent and single input single output functions: y = f(x) References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

14. 2/9/2004Neural Networks14 Training Data Set Adjust weights (w) to learn a given target function: y = f(x) Given a set of training data X→Y References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

15. 2/9/2004Neural Networks15 Training Weights: Error Back-Propagation (BP) Weight update formula: References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

16. 2/9/2004Neural Networks16 Error Back-Propagation (BP) Training error term: e References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

17. 2/9/2004Neural Networks17 BP Formulation References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

18. 2/9/2004Neural Networks18 BP Formulation References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

19. 2/9/2004Neural Networks19 BP Formulation References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

20. 2/9/2004Neural Networks20 BP Formulation References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

21. 2/9/2004Neural Networks21 BP Formulation References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

22. 2/9/2004Neural Networks22 BP Formulation References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

23. 2/9/2004Neural Networks23 BP Formulation References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

24. 2/9/2004Neural Networks24 BP Formulation References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

25. 2/9/2004Neural Networks25 Example: The XOR problem: Single hidden layer: 3 Sigmoid neurons 2 inputs, 1 output Desired I/O table (XOR): x1x2y Example 1000 Example 2011 Example 3101 Example 4110 References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

26. 2/9/2004Neural Networks26 Example: The XOR problem: Training error over epoch References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

27. 2/9/2004Neural Networks27 Example: The XOR problem: initial_weights = 0.0654 0.2017 0.0769 0.1782 0.0243 0.0806 0.0174 0.1270 0.0599 0.1184 0.1335 0.0737 0.1511 final_weights = 4.6970 -4.6585 2.0932 5.5168 -5.7073 -3.2338 -0.1886 1.6164 -0.1929 -6.8066 6.8477 -1.6886 4.1531 References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

28. 2/9/2004Neural Networks28 Example: The XOR problem: Mapping produced by the trained neural net: x1x2y Example 100 0.0824 Example 201 0.9095 Example 310 0.9470 Example 411 0.0464 References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples

29. 2/9/2004Neural Networks29 Example: Overtraining Single hidden layer: 10 Sigmoid neurons 1 input, 1 output References Introduction History Biologically Inspired Applications The Perceptron Activation Functions Hidden Layer Networks Training with BP Examples