1. Artificial Intelligence Neural Networks ( Chapter 9 )

2. Outline of this Chapter Biological Neurons Neural networks History Artificial Neural Network Perceptrons Multilayer Neural Network Applications of neural networks

3. Neural Network A broad class of models that mimic functioning inside the human brain There are various classes of NN models. They are different from each other depending on  Problem types  Structure of the model  Model building algorithm For this discussion we are going to focus on Feed-forward Back-propagation Neural Network (used for Prediction and Classification problems) Definition

4. Biological Neurons The brain is made up of neurons (nerve cells) which have –dendrites (inputs) –a cell body (soma) –an axon (outputs) –synapse (connections between cells) Synapses can be excitatory (potential-increasing activity) or inhibitory (potential-decreasing), and may change over time The synapse releases a chemical transmitter – the sum of which can cause a threshold to be reached – causing the neuron to fire (electrical impulse is sent down the axon )

5. Biology of a neuron

6. Most important functional unit in human brain – a class of cells called – NEURON Dendrites – Receive information A bit of biology... Hippocampal Neurons Source: heart.cbl.utoronto.ca/ ~berj/projects.html Cell Body – Process information Axon – Carries processed information to other neurons Synapse – Junction between Axon end and Dendrites of other Neurons Dendrites Cell Body Axon Schematic Synapse

7. An Artificial Neuron Receives Inputs X 1 X 2 … X p from other neurons or environment Inputs fed-in through connections with ‘weights’ Total Input = Weighted sum of inputs from all sources Transfer function (Activation function) converts the input to output Output goes to other neurons or environment f X1X1 X2X2 XpXp I I = w 1 X 1 + w 2 X 2 + w 3 X 3 +… + w p X p V = f(I) w1w1 w2w2...... wpwp DendritesCell BodyAxon Direction of flow of Information

8. Biological Neurons (cont.) When the inputs reach some threshold an action potential (electrical pulse) is sent along the axon to the outputs. The pulse spreads out along the axon reaching synapse & releasing transmitters into the bodies of other cells. Learning occurs as a result of the synapse’ plasticicity: They exhibit long-term changes in connection strength. There are about 10 11 neurons and about 10 14 synapses in the human brain(!) A neuron may connect to as many as 100,000 other neurons

9. Brain structure We still do not know exactly how the brain works. e.g, born with about 100 billion neurons in our brain. Many die as we progress through life, & are not replaced, but we continue to learn. But we do know certain things about it. Different areas of the brain have different functions –Some areas seem to have the same function in all humans (e.g., Broca’s region- speech & language); the overall layout is generally consistent –Some areas vary in their function; also, the lower-level structure and function vary greatly emotions, reasoning, planning, movement, & parts of speech. senses hearing, memory, meaning, and language vision & ability to recognize objects

10. Brain structure (cont.) We don’t know how different functions are “assigned” or acquired –Partly the result of the physical layout / connection to inputs (sensors) and outputs (effectors) –Partly the result of experience (learning) We really don’t understand how this neural structure/ collection of simple cells leads to action, consciousness and thought Artificial neural networks are not nearly as complex as the actual brain structure

11. Comparing brains with computers They are more neurons in human brain than they are bits in computers Human brain is evolving very slowly---computer memories are growing rapidly. There are a lot more neurons than we can reasonably model in modern digital computers, and they all fire in parallel NN running on a serial computer requires 100 of cycles to decided if a single N will fire-- -in real brain, all Ns do this in a single step. e.g. brain recognizes a face in less than a sec--- billion of cycles Neural networks are designed to be massively parallel The brain is effectively a billion times faster at what it does ComputerHuman Brain Computational units Storage units Cycle time Bandwidth Neuron updates/sec 1 CPU, 10 5 gates 10 9 Bits RAM, 10 11 bits disk 10 -8 Sec 10 9 bits/sec 10 5 10 11 neurons 10 11 neurons, 10 14 synapses 10 -3 sec 10 14 bits/sec 10 14

12. Neural networks History McCulloch & Pitts (1943) are generally recognised as the designers of the first neural network Many of their ideas still used today (e.g. a neuron has a threshold level and once that level is reached the neuron fires is still the fundamental way in which artificial neural networks operate) Hebb (1949) developed the first learning rule (on the premise that if two neurons were active at the same time the strength between them should be increased). During the 50’s and 60’s many researchers such as Minsky & Papert, worked on the perceptron (NNModel) 1969 saw the death of neural network research for about 15 years Only in the mid 80’s (Parker and LeCun) NN research revived.

13. Artificial Neural Network (Artificial) Neural networks are made up of nodes/units connected by links which have –inputs edges, each link has a numeric weight –outputs edges (with weights) –an activation level (a function of the inputs) The computation is split into 2 components: 1.Linear component, called input function (in i )-- computes the weighted sum of the unit’s input values. 2.Non-linear component, called activation function (g)– transforms the weighted sum into the final value that serves as the unit’s activation value: a i = g( in i ) = g(  a j w j,i ) Some nodes are inputs (perception), some are outputs (action)

14. Modeling a Neuron Each unit does a local computation based on inputs from its neighbours & compute a new activation level – sends along each of its output links a j : Activation value of unit j w j,I : Weight on the link from unit j to unit i in I : Weighted sum of inputs to unit i a I : Activation value of unit i g: Activation function.

15. Activation Functions Different models are obtained by using different mathematical functions for g. 3 common choices are: Step(x)=1 if x >= 0, else 0 Sign(x)=+1 if x >= 0, else –1 Sigmoid(x)=1/(1+e -x ) ( in which we try to minimize the error by adjusting the weights of the network, e : represents error degree) 1  represents the firing of a pulse down the axon, & 0 represents no firing. t  (threshold) represents the min total weighted input needed to cause the neuron to fire. threshold function logistic function

16. ANN (Artificial Neural Network) – Feed-forward Network A collection of neurons form a ‘Layer’ Direction of information flow X1X1 X2X2 X3X3 X4X4 y1y1 y2y2 Input Layer - Each neuron gets ONLY one input, directly from outside Output Layer - Output of each neuron directly goes to outside Hidden Layer - Connects Input and Output layers

17. Number of hidden layers can be NoneOne More ANN (Artificial Neural Network) – Feed-forward Network

18. Implementing logical functions McCulloch and Pitts: every Boolean function AND, OR, & NOT can be represented by units with suitable weights & thresholds. We can use these units to build a network to compute any Boolean function (t = threshold or the value of the Bias weight that determines the threshold to cause the neuron to fire)

19. Network Structure They are 2 main categories of NN structure: Feed-forward/acyclic networks:  allow signals to travel one way only; from input to output. There is no feedback (loops).  Tend to be straight forward networks that associate inputs with outputs. (i.e. pattern recognition.)  Usually arranged in layers– each unit receives input only from units in preceding layer, no links between units in the same layer. –single-layer perceptrons –multi-layer perceptrons Recurrent/cyclic networks: –Feeds its outputs back into its own inputs. –recurrent neural nets have directed cycles with delays –The links can form arbitrary topologies. The brain is recurrent network – activation is fed back to the units that caused it.

20. Feed-forward example a5 = g(W3,5.a3 +W4,5.a4) = g(W3,5. g(W1,3.a1 +W2,3.a2) +W4,5 g(W1,4.a1 +W2,4.a2)) By adjusting weights, we change the function that the Network represents: learning occurs in NN this way! Simple NN with 2 inputs, 2 hidden units & 1 output unit. No direct connection to the outside world

21. Perceptron Is a network with all inputs connected directly to the output. This is called a single layer NN (Neural Network) or a Perceptron Network. It is a simple form of NN that is used for classification of linearly separable patterns. (i.e. If we have 2 results we can separate them with a line with each group result on a different side of the line)

22. Perceptron or a Single-layer NN  A Feed-Forward NN with no hidden units.  Output units all operate separately--no shared weights.  First Studied in the 50’s  Other networks were known about but the perceptron was the only one capable of learning and thus all research was concentrated in this area.  A single weight only affects one output so we can limit our study to a model as shown on the right  Notation can be simpler, i.e.

23. Multilayer NN Network with 1/more layers of hidden units Layers are usually fully connected; numbers of hidden units typically chosen by hand

24. Summary Most brains have lots of neurons; each neuron  linear threshold unit (?) Perceptrons (one-layer networks) insufficiently expressive Multi-layer networks are sufficiently expressive; can be trained by gradient descent, i.e., error back-propagation Many applications: speech, driving, handwriting, fraud detection, etc. Engineering, cognitive modelling, and neural system modelling sub fields have largely diverged

25. End of Chapter 9