1. Artificial Intelligence Methods Neural Networks Lecture 1 Rakesh K. Bissoondeeal Rakesh K. Bissoondeeal(rakesh.bissoondeeal@ntu.ac.uk)

2. Biological Neural Networks

3. Biological Neuron Synapses Synapses - Gap between adjacent neurons across which chemicals are transmitted: input - Gap between adjacent neurons across which chemicals are transmitted: input Dendrites Dendrites - Receive synaptic contacts from other neurons - Receive synaptic contacts from other neurons Cell body/Soma Cell body/Soma - Metabolic centre of the neuron: processing - Metabolic centre of the neuron: processing Axon Axon - produces the output - produces the output

4. Artificial Neuron Artificial neurons are the building blocks of Artificial Neural Networks Artificial neurons are the building blocks of Artificial Neural Networks

5. Artificial Neurons Artificial neurons simulate the four basic functions of natural neurons Artificial neurons simulate the four basic functions of natural neurons - Signals are passed between neurons over connection links - Each connection link has an associated weight which multiplies the signal transmitted - Each neuron applies an activation function to is net input (sum of weighted input signals) to produce an output signal

6. Why study Artificial Neural Networks Desire to understand the brain and to imitate some of its strength Desire to understand the brain and to imitate some of its strength Traditional computers implement a sequence of logical and arithmetic operations but don’t have the ability to adapt their structure or learn Traditional computers implement a sequence of logical and arithmetic operations but don’t have the ability to adapt their structure or learn Learn from examples, Generalisation Learn from examples, Generalisation Used to solved task where it is beneficial to use a machine but impossible to program all possible outcomes Used to solved task where it is beneficial to use a machine but impossible to program all possible outcomes

7. Applications List of applications mentioned in the literature List of applications mentioned in the literature Aerospace -high performance aircraft autopilot Aerospace -high performance aircraft autopilot Banking –check and other document reading Banking –check and other document reading Defence – weapon steering Defence – weapon steering Financial –financial analysis Financial –financial analysis Speech – speech recognition Speech – speech recognition

8. Brief History of ANNs 1943 W.S. McCulloch and W. Pitts 1943 W.S. McCulloch and W. Pitts - Original idea published 1949 D. Hebb 1949 D. Hebb - Publishes ideas on learning in biological neurons 1958 F. Rosenblatt 1958 F. Rosenblatt - First practical working networks called perceptrons

9. Brief History of ANNs 1969. M Minsky and S. Papert 1969. M Minsky and S. Papert - Rubbish ANNs - Most research on ANNs stop 1970s Widrow, Parker and others 1970s Widrow, Parker and others - Low level of activity - Backpropagation invented 1980s Rumelhart and others 1980s Rumelhart and others - Rediscovery of Backpropagation - Revival of interest in ANNs

10. McCulloch-Pitts Neuron First mathematical model of the biological neuron First mathematical model of the biological neuron - Mc Culloch and Pitts (1943) Most models used today are descended from McCulloch and Pitts neuron Most models used today are descended from McCulloch and Pitts neuron

11. McCulloch-Pitts Neuron 2 2 X1X1 X2X2 X3X3 Y  The output of a neuron is binary. That is, the neuron either fires (output of one) or does not fire (output of zero).

12. McCulloch-Pitts Neuron 2 2 X1X1 X2X2 X3X3 Y  Neurons in a McCulloch-Pitts network are connected by directed, weighted paths  A connection path is excitatory if the weight on the path is positive; otherwise it is inhibitory

13. McCulloch-Pitts Neuron Each neuron has a fixed threshold (. If the net input to the neuron is greater than the threshold, the neuron fires Each neuron has a fixed threshold ( θ). If the net input to the neuron is greater than the threshold, the neuron fires If net input >= If net input >= θ, output=1 If net input < output = 0 If net input < θ, output = 0 2 2 X1X1 X2X2 X3X3 Y

14. Example 1 Logic Functions: AND Logic Functions: AND True=1, False=0 True=1, False=0 If both inputs true, output true If both inputs true, output true Else, output false Else, output false Threshold(Y)=2 Threshold(Y)=2 x1x2AND 000 010 100 111 AND Function 1 1 X1X1 X2X2 Y

15. Example 2 Logic Functions: OR Logic Functions: OR True=1, False=0 True=1, False=0 If either of inputs true, output true If either of inputs true, output true Else, output false Else, output false Threshold(Y)=2 Threshold(Y)=2 x1x2OR 000 011 101 111 OR Function 2 2 X1X1 X2X2 Y

16. McCulloch-Pitts Neuron Structure does not change Structure does not change - Fixed system that takes inputs to produce output Has no concept of learning Has no concept of learning However, McCulloch-Pitts Neuron forms the foundation of modern ANNs However, McCulloch-Pitts Neuron forms the foundation of modern ANNs - Changes made to allow learning

17. Recommended Reading Fundamentals of neural networks; Architectures, Algorithms and Applications, L. Fausett, 1994. Fundamentals of neural networks; Architectures, Algorithms and Applications, L. Fausett, 1994. Artificial Intelligence: A Modern Approach, S. Russel and P. Norvig, 1995. Artificial Intelligence: A Modern Approach, S. Russel and P. Norvig, 1995. An Introduction to Neural Networks. 2 nd Edition, Morton, IM. An Introduction to Neural Networks. 2 nd Edition, Morton, IM.