1. FCAC, University of Hyderabad 1 / 59 Connections Chapter 6 Paul Thagard (2005). Mind: An Introduction to Cognitive Science. 2 nd Edition. MIT Press

2. FCAC, University of Hyderabad 2 / 59 Outline Connectionist Mental Representations Evaluation of Connectionist Representation scheme –Representational power –Computational power Problem Solving: Planning, Decision, Explanation Learning Language –Psychological Plausibility –Neurological Plausibility –Practical Applicability

3. FCAC, University of Hyderabad 3 / 59

4. FCAC, University of Hyderabad 4 / 59

5. FCAC, University of Hyderabad 5 / 59 Biological Neural Networks Study of Neural Networks originates in biological systems Human Brain: contains over 100 billion neurons, number of synapses is approximately 1000 times that in electronic circuit terms: synaptic fan-in fan-out is 1000, switching time of a neuron is order of milliseconds But on a face recognition problem brain beats fastest supercomputer in terms of number of cycles of computation to arrive at answer Neuronal Structure Cell body Dendrites for input Axon carries output to other dendrites Synapse-where they meet Activation signal (voltage) travels along axon

6. FCAC, University of Hyderabad 6 / 59

7. FCAC, University of Hyderabad 7 / 59 Von Neumann versus Biological Computer

8. FCAC, University of Hyderabad 8 / 59 Need for ANN Standard Von Neumman Computing as existing presently has some shortcomings. Following are some desirable characteristics in ANN Learning Ability Generalization and Adaptation Distributed and Parallel representation Fault Tolerance Low Power requirements Performance comes not just from the computational elements themselves but the manner of networked interconnectedness of the decision process.

9. FCAC, University of Hyderabad 9 / 59 Learning Paradigms Three Basic Paradigms –Supervised Learning: Learning with a Teacher Example NNs: Perceptron, MLP, RBF, SVM –Unsupervised Learning: Learning without a Teacher Examples: SOM, ART, PCA, ICA –Reinforcement Learning: Learning with a Critic Examples: Q-Learning, TD Learning

10. FCAC, University of Hyderabad 10 / 59 Artifical Neuron Working Model Objective is to create a model of functioning of biological neuron to aid computation All signals at synapses are summed i.e. all the excitatory and inhibitory influences and represented by a net value h(.) If the excitatory influences are dominant, then the neuron fires, this is modeled by a simple threshold function  (.) Certain inputs are fixed biases Output y leads to other neurons McCulloch Pitts Model

11. FCAC, University of Hyderabad 11 / 59 More about the Model Activation Functions play a key role –Simple thresholding (hard limiting) –Squashing Function (sigmoid) –Gaussian Function –Linear Function Biases are also learnt

12. FCAC, University of Hyderabad 12 / 59 Different Kinds of Network Architectures

13. FCAC, University of Hyderabad 13 / 59 Major Learning Rules Error Correction: Error signal (d–y) used to adjust the weights so that eventually desired output d is produced Perceptron Solving “AND” Problem

14. FCAC, University of Hyderabad 14 / 59 Major Learning Rules Hebbian:weights are adjusted by a factor proportional to the activities of the neurons associated Orientation Selectivity of a Single Hebbian Neuron

15. FCAC, University of Hyderabad 15 / 59 Major Learning Rules Competitive Learning: “winner take all” (a) Before Learning (b) After Learning

16. FCAC, University of Hyderabad 16 / 59 ANN Applications Pattern Classification –Speech Recognition, ECG/EEG classification, OCR

17. FCAC, University of Hyderabad 17 / 59 ANN Applications Function Approximation –Noisy arbitrary function needs to be approximated

18. FCAC, University of Hyderabad 18 / 59 ANN Applications Prediction/Forecasting – Given a function of time, predict the function values for future time values, used in weather prediction and stock market predictions

19. FCAC, University of Hyderabad 19 / 59 Supervised Learning Rules Widrow–Hoff rule : Correlation rule : Perceptron / Delta rule :

20. FCAC, University of Hyderabad 20 / 59 Perceptron (Rosenblatt, 1962)

21. FCAC, University of Hyderabad 21 / 59 What a Perceptron Does Regression: y = wx+w 0 Classification: y=Sigmoid(wx+w 0 >0) w w0w0 y x x 0 =+1 y x w w0w0 y x s w0w0

22. FCAC, University of Hyderabad 22 / 59 K Outputs Classification : Regression :

23. FCAC, University of Hyderabad 23 / 59 Training Online (instances seen one by one) vs batch (whole sample) learning: –No need to store the whole sample –Problem may change in time –Wear and degradation in system components Generic update rule (LMS rule):

24. FCAC, University of Hyderabad 24 / 59 What does Perceptron do?

25. FCAC, University of Hyderabad 25 / 59

26. FCAC, University of Hyderabad 26 / 59 Learning Boolean AND

27. FCAC, University of Hyderabad 27 / 59

28. FCAC, University of Hyderabad 28 / 59

29. FCAC, University of Hyderabad 29 / 59 XOR No w 0, w 1, w 2 satisfy: (Minsky and Papert, 1969)

30. FCAC, University of Hyderabad 30 / 59 Linearly Nonseparable Patterns Example: XOR function, etc.

31. FCAC, University of Hyderabad 31 / 59

32. FCAC, University of Hyderabad 32 / 59

33. FCAC, University of Hyderabad 33 / 59

34. FCAC, University of Hyderabad 34 / 59 Multilayer Perceptrons (Rumelhart et al., 1986)

35. FCAC, University of Hyderabad 35 / 59 x 1 XOR x 2 = (x 1 AND ~x 2 ) OR (~x 1 AND x 2 )

36. FCAC, University of Hyderabad 36 / 59 Backpropagation

37. FCAC, University of Hyderabad 37 / 59

38. FCAC, University of Hyderabad 38 / 59

39. FCAC, University of Hyderabad 39 / 59

40. FCAC, University of Hyderabad 40 / 59

41. FCAC, University of Hyderabad 41 / 59

42. FCAC, University of Hyderabad 42 / 59

43. FCAC, University of Hyderabad 43 / 59

44. FCAC, University of Hyderabad 44 / 59

45. FCAC, University of Hyderabad 45 / 59 ANN Application: Vehicle Steering

46. FCAC, University of Hyderabad 46 / 59

47. FCAC, University of Hyderabad 47 / 59

48. FCAC, University of Hyderabad 48 / 59

49. FCAC, University of Hyderabad 49 / 59 Computational Power Problem Solving: NNs provide powerful computational tools for performing constraint satisfaction. Likes programming Computer geek Likes parties Outgoing Shy  + + +

50. FCAC, University of Hyderabad 50 / 59 Computational Power (contd.) Planning: How to accomplish a goal from an initial or current state? –Solving CSP in a parallel fashion –Constraints can be satisfied in parallel by repeatedly passing activation among all the units, until they attain stable activation levels (relaxation or settling)

51. FCAC, University of Hyderabad 51 / 59 Computational Power (contd.) Decision: Selection of best means to get to the goal. Similar to planning, decisions based on solving CSP in a parallel fashion Goal Priority Understand moreRetire Early Live Well Learn More Long-term Income Immediate Income Graduate School Corporation  ++ +

52. FCAC, University of Hyderabad 52 / 59 Computational Power (contd.) Explanation: In this, you are trying to understand why something happened. –Explanation can be understood as activation of prototypes encoded in distributed networks. Fred wants high grades Fred is studying Fred likes to party Fred went dancing Fred was seen in the library Fred did not come EVIDENCE 

53. FCAC, University of Hyderabad 53 / 59 Computational Power (contd.) Learning: Ability to use experience to improve performance. Two basic ways learning can take place: –Add new units to the network –Change weights –Various learning rules we discussed, Hebbian learning etc. in the Supervised and Unsupervised modes of learning.

54. FCAC, University of Hyderabad 54 / 59 Computational Power (contd.) Language: How do connections underlie our ability to learn and use language? –Connectionist models of language for visual and auditory perception –Word recognition as parallel CSP Connectionist Network for Word sense disambiguation (WSD) Financial institution State Bank of India boundary Godavari bank1 bank2 moneyriver 

55. FCAC, University of Hyderabad 55 / 59 Psychological Plausibility Psychological plausibility assessed through experiments on how connectionist models implement human behaviour –McClelland and Rumelhart (1981) model of word perception explained results of several psychololinguistic experiments –McClelland and Elman (1986) describe various speech perception phenomena explained by their connectionist model –Kinstsch et al (1990) model explains results on discourse comprehension

56. FCAC, University of Hyderabad 56 / 59 Neurological Plausibility Neural network models have close resemblance to networks of real neurons in the brain. –Local versus distributed representation in the brain discussed earlier.

57. FCAC, University of Hyderabad 57 / 59 Practical Applicability Connectionist models of Reading –Reading as kind of parallel constraint satisfaction where the constraints simultaneously involve spelling, meaning and context. Industrial applications –As discussed earlier Print Orthographic Processor Meaning Processor Context Processor

58. FCAC, University of Hyderabad 58 / 59 References Paul Thagard (2005). Mind: An Introduction to Cognitive Science. 2nd Edition. MIT Press. Chris Eliasmith’s slides from University of Toronto Ethem Alpaydin’s Machine Learning (2004) slides, Chapter 11.

59. FCAC, University of Hyderabad 59 / 59 Further Plan Chapter 8 (Review and Evaluation) PART II: Extensions to Cognitive Science –Chapter 9: Brains Oct 20 (Tue), Pavan / Praveen –Chapter 10: Emotions Oct 22 (Thu) Sreenivas / Suchitra –Chapter 11: Consciousness Oct 27 (Tue) Venkat / Nature vs Nurture fight-out (Kiran / Rakesh) –Chapter 12: Bodies, the world and Dynamic Systems Oct 29 (Thu) Rakesh –Chapter 13: Societies Nov 3 (Tue) Mercy / Shanta –Chapter 14: The Future of CogSci Nov 5 (Thu)