1. 1 Artificial Neural Networks: An Introduction S. Bapi Raju Dept. of Computer and Information Sciences, University of Hyderabad

2. ANN-Intro (Jan 2010) 2 of 29 OUTLINE Biological Neural Networks Applications of Artificial Neural Networks Taxonomy of Artificial Neural Networks Supervised and Unsupervised Artificial Neural Networks Basis function and Activation function Learning Rules Applications OCR, Load Forecasting, Condition Monitoring

3. ANN-Intro (Jan 2010) 3 of 29 Biological Neural Networks Study of Neural Networks originates in biological systems Human Brain: contains over 100 billion neurons, number of synapses is approximately 1000 times that  in electronic circuit terms: synaptic fan-in fan-out is 1000,  switching time of a neuron is order of milliseconds  But on a face recognition problem brain beats fastest supercomputer in terms of number of cycles of computation to arrive at answer Neuronal Structure Cell body Dendrites for input Axon carries output to other dendrites Synapse-where they meet Activation signal (voltage) travels along axon

4. ANN-Intro (Jan 2010) 4 of 29 Need for ANN Standard Von Neumman Computing as existing presently has some shortcomings. Following are some desirable characteristics in ANN Learning Ability Generalization and Adaptation Distributed and Parallel representation Fault Tolerance Low Power requirements Performance comes not just from the computational elements themselves but the manner of networked interconnectedness of the decision process.

5. ANN-Intro (Jan 2010) 5 of 29 Von Neumann versus Biological Computer

6. ANN-Intro (Jan 2010) 6 of 29 ANN Applications Pattern Classification Speech Recognition, ECG/EEG classification, OCR

7. ANN-Intro (Jan 2010) 7 of 29 ANN Applications Clustering/Categorization Data mining, data compression

8. ANN-Intro (Jan 2010) 8 of 29 ANN Applications Function Approximation Noisy arbitrary function needs to be approximated

9. ANN-Intro (Jan 2010) 9 of 29 ANN Applications Prediction/Forecasting Given a function of time, predict the function values for future time values, used in weather prediction and stock market predictions

10. ANN-Intro (Jan 2010) 10 of 29 ANN Applications Optimization Several scientific and other problems can be reduced to an optimization problem like the Traveling Salesman Problem (TSP)

11. ANN-Intro (Jan 2010) 11 of 29 ANN Applications Content Based Retrieval Given the partial description of an object retrieve the objects that match this

12. ANN-Intro (Jan 2010) 12 of 29 ANN Applications Control Model-reference adaptive control, set-point control Engine idle-speed control

13. ANN-Intro (Jan 2010) 13 of 29 Characteristics of ANN Biologically inspired computational units Also called as Connectionist Models or Connectionist Architectures Large number of simple processing elements Very large number of weighted connections between elements. Information in the network is encoded in the weights learned by the connections Parallel and distributed control Connection weights are learned by automatic training techniques

14. ANN-Intro (Jan 2010) 14 of 29 Artifical Neuron Working Model Objective is to create a model of functioning of biological neuron to aid computation All signals at synapses are summed i.e. all the excitatory and inhibitory influences and represented by a net value h(.) If the excitatory influences are dominant, then the neuron fires, this is modeled by a simple threshold function  (.) Certain inputs are fixed biases Output y leads to other neurons McCulloch Pitts Model

15. ANN-Intro (Jan 2010) 15 of 29 More about the Model Activation Functions play a key role Simple thresholding (hard limiting) Squashing Function (sigmoid) Gaussian Function Linear Function Biases are also learnt

16. ANN-Intro (Jan 2010) 16 of 29 Different Kinds of Network Architectures

17. ANN-Intro (Jan 2010) 17 of 29 Learning Ability Mere Architecture is insufficient Learning Techniques also need to be formulated Learning is a process where connection weights are adjusted Learning is done by training from labeled examples. This is the most powerful and useful aspect of neural networks in their use as “Black Box” classifiers. Most commonly an input-output relationship is learnt Learning Paradigm needs to be specified Weight update in learning rules must be specified Learning Algorithm specifies step by step procedure

18. ANN-Intro (Jan 2010) 18 of 29 Learning Theory Major Factors Learning Capacity: This concerns the number of patterns that can be learnt and the functions and kinds of decision boundaries that can be formed Sample Complexity: This concerns the number of the samples needed to learn with generalization. “Overfitting” problem is to be avoided Computational Complexity: This concerns the computation time needed to learn the concepts embedded in the training samples. Generally the computational complexity of learning is high.

19. ANN-Intro (Jan 2010) 19 of 29 Learning Issues

20. ANN-Intro (Jan 2010) 20 of 29 Major Learning Rules Error Correction: Error signal (d–y) used to adjust the weights so that eventually desired output d is produced Perceptron Solving “AND” Problem

21. ANN-Intro (Jan 2010) 21 of 29 Major Learning Rules Error Correction: in Mutlilayer Feedforward Network Geometric interpretation of the role of hidden units in a 2D input space

22. ANN-Intro (Jan 2010) 22 of 29 Major Learning Rules Hebbian:weights are adjusted by a factor proportional to the activities of the neurons associated Orientation Selectivity of a Single Hebbian Neuron

23. ANN-Intro (Jan 2010) 23 of 29 Major Learning Rules Competitive Learning: “winner take all” (a) Before Learning (b) After Learning

24. ANN-Intro (Jan 2010) 24 of 29 Summary of ANN Algorithms

25. ANN-Intro (Jan 2010) 25 of 29

26. ANN-Intro (Jan 2010) 26 of 29 Application to OCR System The main problem in the Handwritten Letter recognition is that characters with variation in thickness shape, rotation and different nature of strokes need to be recognized as of being in the different categories for each letter. Sufficient number of sample training data is required for each character to train the networks A Sample set of characters in the NIST Data

27. ANN-Intro (Jan 2010) 27 of 29 OCR Process

28. ANN-Intro (Jan 2010) 28 of 29 OCR Example (continued) Two schemes shown at right First makes use of the feature extractors Second uses the image pixels directly

29. ANN-Intro (Jan 2010) 29 of 29 References A. K. Jain, J.Mao, K.Mohiuddin, “ANN a Tutorial”, IEEE Computer, 1996 March, pp 31- 44 (Figures and Tables taken from this reference) B. Yegnanarayana, Artificial Neural Networks, Prentice Hall of India, 2001. Y. M. Zurada, Inroduction to Artificial Neural Systems, Jaico, 1999. MATLAB neural networks toolbox and manual