1. PSY105 Neural Networks 2/5 2. “A universe of numbers”

2. Lecture 1 recap We can describe patterns at one level of description that emerge due to rules followed at a lower level of description. Neural network modellers hope that we can understand behaviour by creating models of networks of artificial neurons.

3. Warren McCullock 1943 - First artificial neuron model Warren McCulloch (neurophysiologist) Walter Pitts (mathematician)

4. A simple artificial neuron Threshold Add weight input activation Multiply inputs by weights and add. If the sum is larger than a threshold output 1, otherwise output 0 Threshold logic unit (TLU)

5. 0 1 output activation TLU: the output relation threshold The relation is non-linear – small changes in activation give different changes in the output depending on the initial activation

6. Model neuron function, reminders… Inputs vary, they can be 0 or 1 – Weights change, effectively ‘interpreting’ inputs There is a weight for each input – This can be a +ve number (excitation) or a –ve number (inhibition) – Weights do not change when inputs change Activation = weighted sum of inputs – Activation = input1 x weight1 + input2xweight2 etc If activation>threshold, output = 1, otherwise output=0 – Threshold = 1

7. States, weights & functions States: all the possible combinations of inputs Weights: how each input is multiplied before contributing to the activation of the unit Functions: a way inputs are combined to produce outputs

8. Computing with neurons: identify (1) Input 0 1 Weight 0.7 Activation 0 0.7 Output 0 X input weight output ? Threshold = 1 Act. State 1 State 2

9. Computing with neurons: identity (2) Input 0 1 Weight 1 Activation 0 1 Output 0 1 input weight output ? Threshold = 1 Act. State 1 State 2

10. Question: How could you use these simple neurons (TLUs) to compute the AND function? Input 1 0 1 Input 2 0 1 0 1 Output 0 1

11. Computing with neurons: AND inputs weights output ? Input 1 0 1 Input 2 0 1 0 1 Activation 0 0.5 1 Output 0 1 Threshold = 1, Weight 1 = 0.5, Weight 2 = 0.5 Act. State 1 State 2 State 3 State 4

12. Networks of such neurons are Turing complete 1912 - 1954

13. Semilinear node Squashing function weight input Add activation

14. 0 1 output activation Semilinear node: the output relation (squashing function) threshold