1. Artificial neural networks.
COMP Part I.
Artificial neural networks.

2. of the Artificial Neural Networks.
Topic 3.
Learning Rules
of the Artificial Neural Networks.

3. Hebb’s rule (1949). Hebb conjectured that
a particular type of use-dependent modification of the connection strength of synapses might underlie learning in the nervous system.

4. Hebb’s rule (1949).
Hebb introduced a neurophysiological postulate :
“…When an axon of cell A is near enough to excite a cell B and repeatedly and persistently tales part in firing it, some growth process or metabolic change takes place in one or both cells, such that A’s efficiency as one of the cells firing B, is increased.”

5. Hebb’s rule (1949). The simplest formalisation of Hebb’s rule is to
increase weight of connection at every next instant
in the way:
(1)
where
(2)

6. Hebb’s rule (1949). where (2) here
(1)
where
(2)
here
wjik is the weight of connection at instant k,
wjik+1 is the weight of connection at the following instant k+1,
Dwjik is increment by which the weight of connection is enlarged,
C is positive coefficient which determines learning rate,
aik is input value from the presynaptic neuron at instant k,
Xjk is output of the postsynaptic neuron at the same instant k.

7. Hebb’s rule (1949).
(1)
where
(2)
Thus, the weight of connection changes at the next instant only if both preceding input via this connection and the resulting output simultaneously are not equal to 0.

8. Hebb’s rule (1949).
(1)
where
(2)
Equation (2) emphasises the correlation nature of a Hebbian synapse. It is sometimes referred to as the activity product rule.

9. Hebb’s rule (1949).
(1)
where
(2)
Hebb’s original learning rule (2) referred exclusively to excitatory synapses, and has the unfortunate property that it can only increase synaptic weights, thus washing out the distinctive performance of different neurons in a network, as the connections drive into saturation..

10. Hebb’s rule (1949).
(1)
where
(2)
However, when the Hebbian rule is augmented by a normalisation rule, e.g. keeping constant the total strength of synapses upon a given neuron, it tends to “sharpen” a neuron’s predisposition “without a teacher”, causing its firing to become better and better correlated with a cluster of stimulus patterns.

11. Normalised Hebb’s rule.
(1)
where
(2)
normalisation: (3)
Hebb’s rule plays an important role in studies of ANN algorithms much “younger” than the rule itself, such as unsupervised learning or self-organisation.

12. Normalised Hebb in practice.
Input unit No 1 2 3 4 a 1 w 1 w a 2 2 q X a w 3 3 w 4 a 4

13. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=0 C=1
w01
w02
w03
w04 1 1 1
q =1 X 1 1

14. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=0 C=1
w01
w02
w03
w04 1 1 1
q =1 X 1 1

15. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=0 C=1
w01
w02
w03
w04 1 1 1
q =1 X 1 1

16. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=0 C=1
w01
w02
w03
w04
0.5
0.5
0.5
q =1 X
0.5
0.5

17. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=0 C=1
a01
a02
a03
a04 1
w01
w02
w03
w04
0.5 1
0.5
0.5
q =1 X 1
0.5
0.5

18. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=0 C=1
a01
a02
a03
a04 1
w01
w02
w03
w04
0.5 1
0.5
0.5
q =1 X 1
0.5
0.5

19. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=0 C=1
a01
a02
a03
a04 1
w01
w02
w03
w04
0.5 1
0.5
0.5
q =1 1 1
0.5
0.5

20. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=0 C=1
a01
a02
a03
a04 1
w01
w02
w03
w04
0.5 1
0.5
0.5
q =1 1 1
0.5
0.5

21. Normalised Hebb in practice.
Input unit No 1 2 3 4
a01
a02
a03
a04 1
t=1 C=1
w11
w12
w13
w14
1.5
0.5
1.5
0.5
q =1 X
1.5
0.5

22. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=1 C=1
w11
w12
w13
w14
1.5
0.5
1.5
0.5
q =1 X
1.5
0.5

23. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=1 C=1
w11
w12
w13
w14
1.5
0.5
1.5
0.5
q =1 X
1.5
0.5

24. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=1 C=1
w11
w12
w13
w14
0.67
0.22
0.67
0.22
q =1 X
0.67
0.22

25. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=1 C=1
w11
w12
w13
w14
0.67
0.22
0.67
0.22
q =1 X
0.67
0.22

26. Normalised Hebb in practice.
Input unit No 1 2 3 4
w01
w02
w03
w04
0.5
t=1 C=1
w11
w12
w13
w14
0.67
0.22
0.67
0.22
q =1 X
0.67
Continue…
0.22

27. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=1 C=1
a11
a12
a13
a14 1
w11
w12
w13
w14
0.67
0.22 1
0.67
0.22
q =1 X 1
0.67
0.22

28. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=1 C=1
a11
a12
a13
a14 1
w11
w12
w13
w14
0.67
0.22 1
0.67
0.22
q =1 1 1
0.67
0.22

29. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=1 C=1
a11
a12
a13
a14 1
w11
w12
w13
w14
0.67
0.22 1
0.67
0.22
q =1 1 1
0.67
0.22

30. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=2 C=1
w21
w22
w23
w24
1.67
0.22
1.67
0.22
q =1 X
1.67
0.22

31. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=2 C=1
w21
w22
w23
w24
1.67
0.22
1.67
0.22
q =1 X
1.67
0.22

32. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=2 C=1
w21
w22
w23
w24
1.67
0.22
1.67
0.22
q =1 X
1.67
0.22

33. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=2 C=1
w21
w22
w23
w24
0.70
0.09
0.70
0.09
q =1 X
0.70
0.09

34. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=2 C=1
w21
w22
w23
w24
0.70
0.09
0.70
0.09
q =1 X
0.70
0.09

35. Normalised Hebb in practice.
Input unit No 1 2 3 4
w11
w12
w13
w14
0.67
0.22
t=2 C=1
w21
w22
w23
w24
0.70
0.09
0.70
0.09
q =1 X
0.70
0.09
Continue…

36. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=2 C=1
a21
a22
a23
a24 1
w21
w22
w23
w24
0.70
0.09 1
0.70
0.09
q =1 X 1
0.70
0.09

37. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=2 C=1
a21
a22
a23
a24 1
w21
w22
w23
w24
0.70
0.09 1
0.70
0.09
q =1 1 1
0.70
0.09

38. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=2 C=1
a21
a22
a23
a24 1
w21
w22
w23
w24
0.70
0.09 1
0.70
0.09
q =1 1 1
0.70
0.09

39. Normalised Hebb in practice.
Input unit No 1 2 3 4
a21
a22
a23
a24 1
t=3 C=1
w31
w32
w33
w34
1.70
0.09
1.70
0.09
q =1 X
1.70
0.09

40. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=3 C=1
w31
w32
w33
w34
1.70
0.09
1.70
0.09
q =1 X
1.70
0.09

41. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=3 C=1
w31
w32
w33
w34
1.70
0.09
1.70
0.09
q =1 X
1.70
0.09

42. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=3 C=1
w31
w32
w33
w34
0.71
0.04
0.71
0.04
q =1 X
0.71
0.04

43. Normalised Hebb in practice.
Input unit No 1 2 3 4
w21
w22
w23
w24
0.70
0.09
t=3 C=1
w31
w32
w33
w34
0.71
0.04
0.71
0.04
q =1 X
0.71
0.04
Continue…

44. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=3 C=1
a31
a32
a33
a34 1
w31
w32
w33
w34
0.71
0.04 1
0.71
0.04
q =1 X 1
0.71
0.04

45. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=3 C=1
a31
a32
a33
a34 1
w31
w32
w33
w34
0.71
0.04 1
0.71
0.04
q =1 1 1
0.71
0.04

46. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=3 C=1
a31
a32
a33
a34 1
w31
w32
w33
w34
0.71
0.04 1
0.71
0.04
q =1 1 1
0.71
0.04

47. Normalised Hebb in practice.
Input unit No 1 2 3 4
a31
a32
a33
a34 1
t=3 C=1
w31
w32
w33
w34
1.71
0.04 1
1.71
0.04
q =1 1 1
1.71
0.04

48. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=4 C=1
w41
w42
w43
w44
1.71
0.04
1.71
0.04
q =1 X
1.71
0.04

49. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=4 C=1
w41
w42
w43
w44
1.71
0.04
1.71
0.04
q =1 X
1.71
0.04

50. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=4 C=1
w41
w42
w43
w44
0.71
0.02
0.71
0.02
q =1 X
0.71
0.02

51. Normalised Hebb in practice.
Input unit No 1 2 3 4
w31
w32
w33
w34
0.71
0.04
t=4 C=1
w41
w42
w43
w44
0.71
0.02
0.71
0.02
q =1 X
0.71
0.02
Continue…

52. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=4 C=1
a41
a42
a43
a44 1
w41
w42
w43
w44
0.71
0.02 1
0.71
0.02
q =1 X 1
0.71
0.02

53. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=4 C=1
a41
a42
a43
a44 1
w41
w42
w43
w44
0.71
0.02 1
0.71
0.02
q =1 1 1
0.71
0.02

54. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=4 C=1
a41
a42
a43
a44 1
w41
w42
w43
w44
0.71
0.02 1
0.71
0.02
q =1 1 1
0.71
0.02

55. Normalised Hebb in practice.
Input unit No 1 2 3 4
a41
a42
a43
a44 1
t=4 C=1
w51
w52
w53
w54
1.71
0.02 1
1.71
0.02
q =1 1 1
1.71
0.02

56. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=5 C=1
w51
w52
w53
w54
1.71
0.02
1.71
0.02
q =1 X
1.71
0.02

57. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=5 C=1
w51
w52
w53
w54
1.71
0.02
1.71
0.02
q =1 X
1.71
0.02

58. Normalised Hebb in practice.
Input unit No 1 2 3 4
t=5 C=1
w51
w52
w53
w54
0.71
0.01
0.71
0.01
q =1 X
0.71
0.01

59. Normalised Hebb in practice.
Input unit No 1 2 3 4
w41
w42
w43
w44
0.71
0.02
t=5 C=1
w51
w52
w53
w54
0.71
0.01
0.71
0.01
q =1 X
0.71
0.01
STOP!!!!

60. Normalised Hebb in practice.
Input unit No 1 2 3 4 w
1.0 w
1,3
0.5
0.71 w
Iteration N
2,4
0.01
0.0 1 2 3 4 5
q =1
0.3 X
0.71
0.2
0.1
0.01
Iteration N
0.0 1 2 3 4 5

61. Normalised Hebb in practice.
Input unit No 1 2 3 4
Test a1 a2 a3 a4 1 w1 w2 w3 w4
0.71
0.01 1
0.71
0.01 1
q =1 X
0.71
I do not know you…
0.01