1. Other Classification Models: Neural Network
COMP5331
Other Classification Models: Neural Network
Prepared by Raymond Wong
Some of the notes about Neural Network are borrowed from LW Chan’s notes Presented by Raymond Wong
COMP5331

2. Other Classification Models
Support Vector Machine (SVM)
Neural Network
COMP5331

3. Neural Network Neural Network Other terminologies:
A computing system made up of simple and highly interconnected processing elements
Other terminologies:
Connectionist Models
Parallel distributed processing models (PDP)
Artificial Neural Networks
Computational Neural Networks
Neurocomputers
COMP5331

4. Neural Network
This approach is inspired by the way that brains process information, which is entirely differ from the way that conventional computers do
Information processing occurs at many identical and simple processing elements called neurons (or also called units, cells or nodes)
Interneuron connection strengths known as synaptic weights are used to store the knowledge
COMP5331

5. Advantages of Neural Network
Parallel Processing – each neuron operates individually
Fault tolerance – if a small number of neurons break down, the whole system is still able to operate with slight degradation in performance.
COMP5331

6. Neuron Network Neuron Network for OR x1 x2 y 1 OR Function input x11
Neuron Network for OR
input
OR Function
Neuron Network x1
output y x2
COMP5331

7. Neuron Network Neuron Network for OR x1 x2 y 1 OR Function input1
Neuron Network for OR
input
Neuron
OR Function x1
output y x2
COMP5331

8. Neuron Network Neuron Network for OR x1 x2 y 1 input Front Back1
Neuron Network for OR
input
Front
Back
OR Function x1
output
net y x2
COMP5331

9. Neuron Network Neuron Network for OR x1 x2 y 1 input Front Back1
Neuron Network for OR
input
Front
Back
OR Function w1 x1
output
net w2 y x2
Weight
net = w1x1 + w2x2 + b
COMP5331

10. Neuron Network Neuron Network for OR x1 x2 y 1 input Front Back1
Neuron Network for OR
input
Front
Back
OR Function w1 x1
output
net w2 y x2
Activation function
Linear function: y = net or y = a . net
Non-linear function
COMP5331

11. Activation Function Non-linear functions
Threshold function, Step Function, Hard Limiter
Piecewise-Linear Function
Sigmoid Function
Radial Basis Function
COMP5331

12. Threshold function, Step Function, Hard Limiter1
if net 0
y =
if net <0 y 1
net
COMP5331

13. Piecewise-Linear Function1
if net a
y =
½(1/a x net + 1)
if –a < net < a
if net  -a y 1
net -a a
COMP5331

14. Sigmoid Function 1
y =
1 + e-net y 1
net
COMP5331

15. Radial Basis Function
-net2
y = e y 1
net
COMP5331

16. Neuron Network Neuron Network for OR x1 x2 y 1 input Front Back1
Neuron Network for OR
input
Front
Back
OR Function w1 x1
output
net w2 y x2
Threshold function 1
if net 0
net = w1x1 + w2x2 + b
y =
if net <0
COMP5331

17. Learning Let  be the learning rate (a real number)
Learning is done by
wi  wi + (d – y)xi
where
d is the desired output
y is the output of our neural network
b  b + (d – y)
COMP5331

18. Neuron Network Neuron Network for OR x1 x2 d 1 y = if net 0
Neuron Network
net = w1x1 + w2x2 + b x1 x2 d 1
Neuron Network for OR
input
Front
Back
OR Function w1 x1
output
net w2 y x2
Threshold function 1
if net 0
net = w1x1 + w2x2 + b
y =
if net <0
COMP5331

19. Neuron Network x1 x2 d 1 b w1 w2 y = if net 0 if net <0 1
Neuron Network
net = w1x1 + w2x2 + b x1 x2 d 1
 = 0.8
net = w1x1 + w2x2 + b =1
y = 1
Incorrect! b w1 w2
w1 = w1 + (d – y)x1 1 1 1
= 1+0.8*(0-1)*0
= 1
w2 = w2 + (d – y)x2
0.2 1 1
= 1+0.8*(0-1)*0
= 1
b = b + (d – y)
= 1+0.8*(0-1)
= 0.2
COMP5331

20. Neuron Network x1 x2 d 1 b w1 w2 y = if net 0 if net <0 1
Neuron Network
net = w1x1 + w2x2 + b x1 x2 d 1
 = 0.8
net = w1x1 + w2x2 + b
=1.2
y = 1
Correct! b w1 w2
w1 = w1 + (d – y)x1
0.2 1 1
= 1+0.8*(1-1)*0
= 1
w2 = w2 + (d – y)x2
0.2 1 1
= 1+0.8*(1-1)*1
= 1
b = b + (d – y)
= *(1-1)
= 0.2
COMP5331

21. Neuron Network x1 x2 d 1 b w1 w2 y = if net 0 if net <0 1
Neuron Network
net = w1x1 + w2x2 + b x1 x2 d 1
 = 0.8
net = w1x1 + w2x2 + b
=1.2
y = 1
Correct! b w1 w2
w1 = w1 + (d – y)x1
0.2 1 1
= 1+0.8*(1-1)*1
= 1
w2 = w2 + (d – y)x2
0.2 1 1
= 1+0.8*(1-1)*0
= 1
b = b + (d – y)
= *(1-1)
= 0.2
COMP5331

22. Neuron Network x1 x2 d 1 b w1 w2 y = if net 0 if net <0 1
Neuron Network
net = w1x1 + w2x2 + b x1 x2 d 1
 = 0.8
net = w1x1 + w2x2 + b
=2.2
y = 1
Correct! b w1 w2
w1 = w1 + (d – y)x1
0.2 1 1
= 1+0.8*(1-1)*1
= 1
w2 = w2 + (d – y)x2
0.2 1 1
= 1+0.8*(1-1)*1
= 1
b = b + (d – y)
= *(1-1)
= 0.2
COMP5331

23. Neuron Network x1 x2 d 1 b w1 w2 y = if net 0 if net <0 1
Neuron Network
net = w1x1 + w2x2 + b x1 x2 d 1
 = 0.8
net = w1x1 + w2x2 + b
=0.2
y = 1
Incorrect! b w1 w2
w1 = w1 + (d – y)x1
0.2 1 1
= 1+0.8*(0-1)*0
= 1
w2 = w2 + (d – y)x2
-0.6 1 1
= 1+0.8*(0-1)*0
= 1
b = b + (d – y)
We repeat the above process until
the neural networks output the correct values
of y (i.e., y = d for each possible input)
= *(0-1)
= -0.6
COMP5331

24. Neuron Network Neuron Network for OR x1 x2 y 1 input Front Back1
Neuron Network for OR
input
Front
Back
OR Function w1 x1
output
net w2 y x2
Threshold function 1
if net 0
net = w1x1 + w2x2 + b
y =
if net <0
COMP5331

25. Limitation
It can only solve linearly separable problems
COMP5331

26. Multi-layer Perceptron (MLP)
input
Neuron Network x1
output y x2
input
Neuron x1
output y x2
COMP5331

27. Multi-layer Perceptron (MLP)
input
Neuron Network x1
output y x2
input
output x1 y x2
COMP5331

28. Multi-layer Perceptron (MLP)
input
output x1 y1 x2 y2 x3 y3 x4 y4 x5
COMP5331
Input layer
Hidden layer
Output layer

29. Advantages of MLP Can solve A universal approximator
Linear separable problems
Non-linear separable problems
A universal approximator
MLP has proven to be a universal approximator, i.e., it can model all types function y = f(x)
COMP5331