1. Introduction to Connectionism Jaap Murre Universiteit van Amsterdam en Universiteit Utrecht murre@psy.uva.nl

2. Neural networks Based on an abstract view of the neuron Artificial neurons are connected to form large networks The connections determine the function of the network Connections can often be formed by learning and do not need to be ‘programmed’

3. Overview Biological and connectionist neurons –McCulloch and Pitts Hebbian learning –The Hebb rule –Competitive learning –Willshaw networks Illustration of key characteristics –Pattern completion –Graceful degradation

4. Many neurons have elaborate arborizations

5. The axon is covered with myelin sheaths for faster conductivity

6. With single-cell recordings, action potentials (spikes) can be recorded

7. McCulloch-Pitts (1943) Neuron 1. The activity of the neuron is an “all-or- none” process 2. A certain fixed number of synapses must be excited within the period of latent addition in order to excite a neuron at any time, and this number is independent of previous activity and position of the neuron

8. McCulloch-Pitts (1943) Neuron 3. The only significant delay within the nervous system is synaptic delay 4. The activity of any inhibitory sysapse absolutely prevents excitation of the neuron at that time 5. The structure of the net does not change with time From: A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5, 115-133.

9. Neural networks abstract from the details of real neurons Conductivity delays are neglected An output signal is either discrete (e.g., 0 or 1) or it is a real-valued number (e.g., between 0 and 1) Net input is calculated as the weighted sum of the input signals Net input is transformed into an output signal via a simple function (e.g., a threshold function)

10. Artificial ‘neuron’

11. How to ‘program’ neural networks? The learning problem Selfridge (1958): evolutionary or ‘shake- and-check’ (hill climbing) Other approaches –Unsupervised or regularity detection –Supervised learning –Reinforcement learning has ‘some’ supervision

12. Neural networks and David Marr’s model (1969) Marr’s ideas are based on the learning rule by Donald Hebb (1949) Hebb-Marr networks can be auto- associative or hetero-associative The work by Marr and Hebb has been extremely influential in neural network theory

13. Hebb (1949) “When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased” From: The organization of behavior.

14. Hebb (1949) Also introduces the word connectionism “The theory is evidently a form of connectionism, one of the switchboard variety, though it does not deal in direct connections between afferent and efferent pathways: not an ‘S-R’psychology, if R means a muscular response. The connections server rather to establish autonomous central activities, which then are the basis of further learning” (p.xix)

15. William James (1890) Let us assume as the basis of all our subsequent reasoning this law: When two elementary brain-processes have been active together or in immediate succession, one of them, on re-occurring, tends to propagate its excitement into the other. From: Psychology (Briefer Course).

16. The Hebb rule is found with long term potentiation (LTP) in the hippocampus

17. With Hebbian learning, two learning methods are possible With unsupervised learning there is no teacher: the network tries to discern regularities in the input patterns With supervised learning an input is associated with an output –If the input and output are the same, we speak of auto-associative learning –If they are different it is called hetero- associative learning

18. Another type of network is based on error-correcting learning The most popular algorithm is called backpropagation

19. We will look at an example of each type of Hebbian learning Competitive learning is a form of unsupervised learning Hetero-associative learning in Willshaw networks is an example of supervised learning

20. Example of competitive learning: Stimulus ‘at’ is presented ato 12

21. Example of competitive learning: Competition starts at category level ato 12

22. Example of competitive learning: Competition resolves ato 12

23. Example of competitive learning: Hebbian learning takes place ato 12 Category node 2 now represents ‘at’

24. Presenting ‘to’ leads to activation of category node 1 ato 12

25. ato 12

26. ato 12

27. ato 12

28. Category 1 is established through Hebbian learning as well ato 12 Category node 1 now represents ‘to’

29. Willshaw networks A weight either has the value 0 or 1 A weight is set to 1 if input and output are 1 At retrieval the net input is divided by the total number of active nodes in the input pattern

30. Example of a simple heteroassociative memory of the Willshaw type 1 1 0 1 0 0 000111000111 1 1 1 1 1 1 1 1 1 101010101010 001011001011 0 0 1 0 1 1 1 0 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

31. Example of pattern retrieval 1 1 1 001011001011 1 1 1 1 1 1 1 1 1 3 2 2 3 3 2 1 0 0 1 1 0 Sum = 3 Div by 3 = (1 0 0 1 1 0)

32. Example of successful pattern completion using a subpattern 1 1 1 001001001001 1 1 1 1 1 1 1 1 1 2 1 1 2 2 1 1 0 0 1 1 0 Sum = 2 Div by 2 = (1 0 0 1 1 0) 1

33. Example graceful degradation: small lesions have small effects 1 1 1 001011001011 1 1 1 1 1 1 1 1 1 3 2 1 2 3 1 1 0 0 0 1 0 Sum = 3 Div by 3 = (1 0 0 1 1 0)

34. Summing up Neural networks can learn input-output associations They are able to retrieve an output on the basis of incomplete input cues They show graceful degradation Eventually the network will become overloaded with too many patterns