1. Human Visual System Neural Network
Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and Robert Zucker

2. The Visual System Hubel and Wiesel
1981 Nobel Prize for work in early 1960s
Cat’s visual cortex
cats anesthetized, eyes open with controlling muscles paralyzed to fix the stare in a specific direction
thin microelectrodes measure activity in individual cells
cells specifically sensitive to line of light at specific orientation
Key discovery – line and edge detectors in the visual cortex of mammals

3. The Study Compare Two Neural Networks Objective
One without vertical and horizontal line detectors
One with vertical and horizontal line detectors
Objective
Show that the neural network with line detectors is superior to the one without on the six vertical- horizontal line-segment letters E, F, H, I, L, T
Also, experiment with the full alphabet
Without line detectors

4. Uppercase 5x7 Bit-map Alphabet Horizontal-vertical line-segment letters are E, F, H, I, L, T

5. Neural Network Without Line Detectors

6. Neural Network Specification Without Line Detectors
Layers
Input layer: 20x20 retina of binary units
Hidden layer: 50 units (other numbers explored)
Output layer: 6 units for letters E, F, H, I, L, T
Weights
20,000 (400x50) between input and hidden layer
300 (50x6) between hidden and output layer
Total of 20,300 variable weights, no fixed weights

7. Neural Network With Line Detectors

8. Neural Network Specification With Vertical and Horizontal Line Detectors
Layers
Input layer: 20x20 retina of binary units
576 simple vertical and horizontal line detectors
48 complex vertical and horizontal line detectors
Hidden layer: 50 units (other numbers explored)
Output layer: 6 units for letters E, F, H, I, L, T
Weights
6336 (576x11) fixed weights from input to simple detectors
576 fixed weights from simple and complex detectors
2400 (48x50) variable weights from complex detectors to hidden layer
300 (50x6) variable weights from hidden to output layer
Total of 6912 fixed weights and 2700 variable weights

9. Vertical Line Detectors
DETECTORS OVERLAP COVERING EACH POSSIBLE RETINAL POSITION FOR A TOTAL OF 288 (18x16) VERTICAL LINE DETECTORS EACH DETECTOR HAS 5 EXCITATORY AND 6 INHIBITORY INPUTS (11 FIXED WEIGHTS), WITH A THRESHOLD OF 3
Horizontal Line Detectors are Similar

10. Retina Image – Letter “E” in Upper Left Area
Region of possible upper-left corners is shown in green.

11. Retina Image – Letter “E” in Upper Right Area
Region of possible upper-left corners is shown in green.

12. Retina Image – Letter “E” in Lower Right Area
Region of possible upper-left corners is shown in green.

13. Example of Vertical Line Detector on Line Segment of “E” – Detector Activated

14. Example of Shifted Vertical Line Detector on Letter “E” – Detector Not Activated

15. Example of Shifted Vertical Line Detector on Letter “E” – Detector Not Activated

16. 24 Vertical Complex Line Detector Regions Any Simple Line Detector in a Region Activates the Complex Line Detector

17. 24 Horizontal Complex Line Detector Regions Any Simple Line Detector in a Region Activates the Complex Line Detector

18. The Corresponding 48 Complex Horizontal and Vertical Line Detectors
Complex Horizontal and Vertical Line Detector Matrix

19. Experiments Experiment 1 Experiment 2
6 Line-Segment Letters without Line Detectors
26 Letters without Line Detectors
Experiment 2
6 Line-Segment Letters with Line Detectors

20. Experimental Parameter Combinations
Epochs: 50
100
200
400
800
1600
(occasionally)
Hidden Layer Units: 10
18* 50
100
200 *
300*
500*
*Selected cases
Noise: 0% 2% 5%
10%
15%
20%

21. Simulation View – Peltarion’s Synapse Product
Experiment 1

22. Simulation Settings Experiment 2 – Line Detectors 6 Line-Segment Letters: E, F, H, I, L, T
Function Layer:
Function: Tanh Sigmoid
Forward Rule: No rule
Back Rule: Levenberg- Marquardt
Propagator: Function Layer
Weight Layer:
Forward Rule: No rule
Back Rule: Levenberg- Marquardt
Propagator: Weight Layer

23. Exp 1 – 6 Letters, No Line Detectors – 35.42% Accuracy
6 Letters: no line detectors 50 Hidden layer units 50 Epochs 0% noise

24. Exp 1 – 6 Letters, No Line Detectors – 36.25% Accuracy
6 Letters: no line detectors 50 Hidden layer units 1600 Epochs 0% noise

25. Exp 2 – 6 Letters, With Line Detectors – 67.5% Accuracy
6 Letters: with line detectors 50 Hidden layer units 50 Epochs 0% noise

26. Exp 2 – 6 Letters, With Line Detectors – 67.5% Accuracy
6 Letters: with line detectors 50 Hidden layer units 50 Epochs 0% noise

27. Exp 1 – 6 Letters, No Line Detectors – 27.69% Accuracy
6 Letters: no line detectors 10 Hidden layer units 1600 Epochs 0% noise

28. Exp 1 – 6 Letters, With Line Detectors – 82.5% Accuracy
6 Letters: with line detectors 10 Hidden layer units 1600 Epochs 0% noise

29. Exp 2 – 6 Letters, With Line Detectors – 82.5% Accuracy
6 Letters: with line detectors 10 Hidden layer units 1600 Epochs 0% noise

30. Exp 1 – 26 Letters, No Line Detectors – 27.69% Accuracy
26 Letters: no line detectors 50 Hidden layer units 1600 Epochs 0% noise

31. Exp 1 – 26 Letters, No Line Detectors – 27.69% Accuracy
26 Letters: no line detectors 50 Hidden layer units 1600 Epochs 0% noise

32. Exp 1 – 6 Letters, No Line Detectors Epochs versus Percent Added Noise
Percent Noise
Epochs 0% 2% 5%
10%
15%
20% 50
35.42
20.83
24.17 20
20.42
100
36.25
21.67 25
18.75
200
23.75
400
22.50
800
36.67
20.00
1600
18.33

33. Exp 1 – 26 Letters, No Line Detectors Epochs versus Percent Added Noise
Percent Noise
Epochs 0% 2% 5%
10%
15%
20% 50
23.08
8.17
7.21
7.50
6.25
6.83
100
23.56
9.04
7.69
7.31
6.15
4.62
200
25.00
9.81
7.60
5.58
6.06
4.52
400
27.12
6.92
5.02
5.87
800
28.37
8.96
5.48
5.19
1600
27.69
10.10
5.67

34. Exp 2 – 6 Letters, With Line Detectors Epochs versus Percent Added Noise0% 2% 5%
10%
15%
20% 50
67.50
58.75
46.25
56.67
16.67
42.50
100
72.92
59.17
17.08
28.00
200
77.08
59.58
50.00
62.50
48.75
43.75
400
75.83
58.33
62.92
34.58
28.33
800
78.33
60.42
52.08
32.92
57.08
1600
83.33
60.00
35.00
32.08
3200
85.00
66.67
57.92
52.50

35. Comparison of Line / No-Line Detector Networks 6 letters, 50 hidden layer units, 1600 epochs, no noise
Experiment
Performance
Fixed Weights
Variable Weights
Total Weights
No Line Detectors
36.3%
20,300
Line Detectors
83.3%
6,912
2,700
9,612

36. Main Conclusion
Character recognition performance and efficiency of the neural network using Hubel-Wiesel-like line detectors in the early layers is superior to that of a network using adjustable weights directly from the retina
Recognition performance more than doubled
Line detector network was much more efficient
order of magnitude fewer variable weights and half as many total weights
training time decrease of several orders of magnitude

37. Additional Conclusions
Increasing the number of hidden layer units does not translate to better accuracy, it actually reduces it.
Increasing the number of epochs increased the accuracy but not always
For Experiment 2 (6 letters with line detectors) we can achieve perfect training accuracy and very good validation accuracy
Training time varied from a few minutes to many hours with Experiment 1 – 26 Letters taking the longest out of all, i.e. for 500 hidden layer units it required up to 9 hours.
When noise is added to the retina image the accuracy of the system drops significantly, even for Experiment 2 with the line detectors