1. Pattern Recognition: Neural Networks & Other Methods
Charles Tappert
Seidenberg School of CSIS, Pace University

2. Agenda Neural Network Definitions Linear Discriminant Functions
Simple Two-layer Perceptron
Multilayer Neural Networks
Example Multilayer Neural Network Study
Non Neural Network Pattern Reco Methods

3. Neural Network Definitions
An artificial neural network (ANN) consists of artificial neuron units (threshold logic units) with weighted interconnections
A Perceptron is a term created by Frank Rosenblatt in the late 1950s for an ANN
Unfortunately, the term Perceptron is often misconstrued to mean only a simple two-layer ANN
Therefore, we use the term “simple Perceptron” when referring to a simple two-layer ANN

4. Linear Discriminant Functions
Linear functions of parameters (e.g., features)
The product of an input vector and a weight vector
Hyperplane decision boundaries
Methods of solution
Simple two-layer Perceptron
One weight set connecting input to output units
Some simple problems unsolvable, e.g. XOR
Solve linear algebra directly
Support Vector Machines (SVM)

5. Simple Perceptron
Two-category case (one output unit)

6. Simple Perceptron yields
linear decision boundary

7. Multilayer Neural Networks
Overcome limitations of two-layer networks
Feedforward networks – backpropagation training
A standard 3-layer neural network has an input layer, a hidden layer, and an output layer interconnected by modifiable weights represented by links between layers
Benefits
Simplicity of learning algorithm
Ease of model selection
Incorporation of heuristics/constraints

8. Standard Feedforward Artificial Neural Network (ANN)
The inputs can be raw data or feature data.

9. 3-layer Perceptron solves the XOR problem
(red & black dots below)
Note that each hidden unit acts like a simple Perceptron

11. Perceptrons
Rosenblatt described many types of Perceptrons in his 1962 book Principles of Neurodynamics
Standard 3-layer feedforward Perceptrons
Sufficient to solve any pattern separation problem
Multi-layered (more than 3 layers) Perceptrons
Cross-coupled Perceptrons
Information can flow between units within a layer
Back-coupled Perceptrons
Backward information flow (feedback) between some layers
See

12. Example Neural Network Study: Human Visual System Model
Background
Line and edge detectors are known to exist in the visual systems of mammals (Hubel & Wiesel, Nobel Prize 1981)
See
Problem
Demonstrate on a visual pattern recognition task that an ANN with line detectors is superior to those without line detectors
Hypotheses
A line-detector ANN is more accurate than a non-line-detector ANN
A line-detector ANN trains faster than a non-line-detector ANN
A line-detector ANN requires fewer weights (and especially fewer trainable weights) than a non-line-detector ANN

13. Example Neural Network Study: Human Visual System Model
Introduction – make a case for the study
The Visual System
Biological Simulations of the Visual System
ANN approach to visual pattern recognition
ANNs Using Line and/or Edge Detectors
Current Study
Methodology
Experimental Results
Conclusions and Future Work

14. The Visual System The Visual System Pathway Hubel and Wiesel
Eye, optic nerve, lateral geniculate nucleus, visual cortex
Hubel and Wiesel
1981 Nobel Prize for work in early 1960s
Cat’s visual cortex
cats anesthetized, eyes open with controlling muscles paralyzed to fix the stare in a specific direction
thin microelectrodes measure activity in individual cells
cells specifically sensitive to line of light at specific orientation
Key discovery – line and edge detectors

15. Biological Simulations of Visual System Computational Neuroscience
The Hubel-Wiesel discoveries were instrumental in the creation of what is now called computational neuroscience
Which studies brain function in terms of information processing properties of structures that make up the nervous system
Creates biologically detailed models of the brain
November 2009 – IBM announced they created the largest brain simulation to date on the Blue Gene supercomputer
A billion neurons and trillions of synapses exceeding those in the cat’s brain

16. Artificial Neural Network Approach
Machine learning scientists have taken a different approach to visual pattern recognition using simpler neural network models called ANNs
The most common type of ANN used in pattern recognition is a 3-layer feedforward ANN
Input layer
Hidden layer
Output layer

17. Standard Feedforward Artificial Neural Network (ANN)
The inputs can be raw data or feature data.

18. Literature review of ANNs using line/edge detectors
GIS images/maps – line and edge detectors in four orientations – 0°, 45°, 90°, and 135°
Synthetic Aperture Radar (SAR) images – line detectors constructed from edge detectors
Line detection can be done using edge techniques such as Sobel, Prewitt, Laplacian Gaussian, Zero Crossing and Canny edge detector

19. Current Visual System Study
Use ANNs to simulate line detectors known to exist in the human visual cortex
Construct two feedforward ANNs – one with line detectors and one without – and compare their accuracy and efficiency on a character recognition task
Demonstrate superior performance using pre-wired line detectors

20. Visual System Study – Methodology
Character recognition task - classify straight line uppercase alphabetic characters
Experiment 1 – ANN without line detectors
Experiment 2 – ANN with line detectors
Compare performance
Recognition accuracy
Efficiency – training time & number of weights

21. Alphabetic Input Patterns Six Straight Line Characters (5 x 7 bit patterns)
***** ***** * * * * *****
* * * * * * *
**** **** ***** * * *
***** * * * * ***** *

22. Experiment 1 - ANN without line detectors

23. Experiment 1 - ANN without line detectors
Alphabet character can be placed in any position inside the 20x20 retina not adjacent to an edge – 168 (12*14) possible positions
Training – choose 40 random non-identical positions for each of the 6 characters (~25% of patterns)
Total of 240 (40 x 6) input patterns
Cycle through the sequence E, F, H, I, L, T forty times for one pass (epoch) of the 240 patterns
Testing – choose another 40 random non-identical positions of each character for a total of 240

24. Input patterns on the retina E(2,2) and E(12,5)

25. Experiment 2 - ANN with line detectors

26. Simple horizontal and vertical line detectors
Horizontal Vertical +
288 horizontal and 288 vertical line detectors (a total of 576 simple line detectors)
cover the internal retinal area

27. 24 complex vertical line detectors and their feeding 12 simple line detectors

28. Results – No Line Detectors 10 hidden-layer units: 27.7%
Epochs
Training
Time
Accuracy
Testing 50
~2.5 hr
100%
26.7%
100
~4 hr
28.3%
200
~8 hr
28.8%
400
~16 hr
30.4%
800
~30 hr
1600
~2 days
23.8%
Average
27.7%

29. Results – Line Detectors 10 hidden-layer units: 57.5%
Epochs
Training
Time
Accuracy
Testing 50
0:37 min
47.5%
37.5%
100
0:26 min
100.0%
63.3%
200
0:51 min
68.8%
400
2:28 min
71.3%
50.8%
800
3:37 min
67.9%
1600
8:42 min
95.8%
56.7%
Average
85.8%
57.5%

30. Line Detector Results 50 hidden-layer units: 72.1%
Epochs Set/
Attained
Training
Time
Accuracy
Testing
50/8
41 sec
100%
70.0%
100/9
45 sec
69.8%
200/10
48 sec
71.9%
400/10
49 sec
77.1%
800/8
72.5%
1600/9
71.3%
Average
72.1%

31. Confusion Matrix Overall Accuracy of 77.1%
Out In E F H I L T
62.5 20 5
12.5 80
2.5
7.5 85 95 15
72.5 10
67.5

32. Example Study Conclusion Recognition Accuracy

33. Example Study Conclusion Efficiency of Training Time
ANN with line detectors resulted in a significantly more efficient network
training time decreased by several orders of magnitude

34. Example Study Conclusion Efficiency of Number of Weights
Experiment
Fixed Weights
Variable Weights
Total Weights
1 No Line Detectors
20,300
2 Line Detectors
6,912
2,700
9,612

35. Example Study Overall Conclusions
The strength of the study was its simplicity
The weakness was also it simplicity and that the line detectors appear to be designed specifically for the patterns to be classified
Weakness can be corrected in future work
Add edge detectors
Extend alphabet to full 26 uppercase letters
Add noise to the patterns

36. Non Neural Network Methods
Stochastic methods
Nonmetric methods
Unsupervised learning (clustering)

37. Stochastic Methods Relies of randomness to find model parameters
Used for highly complex problems where gradient descent algorithms unlikely to work
Methods
Simulated annealing
Boltzman learning
Genetic algorithms

38. Nonmetric Methods Nominal data Methods
No measure of distance between vectors
No notion of similarity or ordering
Methods
Decision trees
Grammatical methods
e.g., finite state machines
Rule-based systems
e.g., propositional logic or first-order logic

39. Unsupervised Learning
Often called clustering
The system is not given a set of labeled patterns for training
Instead, the system itself establishes the classes based on the regularities of the patterns

40. Clustering Separate Clouds
Methods work fine when clusters form well separated compact clouds
Less well when there are great differences in the number of samples in different clusters

41. Hierarchical Clustering
Sometimes clusters are not disjoint, but may have subclusters, which in turn having sub-subclusters, etc.
Consider partitioning n samples into clusters
Start with n cluster, each one containing exactly one sample
Then partition into n-1 clusters, then into n-2, etc.

42. Dendrogram of uppercase A’s from DPS Dissertation by Dr. Mary Manfredi

43. Pattern Recognition DPS Dissertations (parentheses indicate in progress)
Visual Systems – Rick Bassett, Sheb Bishop, Tom Lombardi
Speech Recognition – Jonathan Law
Handwriting Recognition – Mary Manfredi
Natural Language Processing – Bashir Ahmed, (Ted Markowitz)
Neural Networks – (John Casarella, Robb Zucker)
Keystroke Biometric – Mary Curtin, Mary Villani
Stylometry Biometric – (John Stewart)
Fundamental Research – Kwang Lee, Carl Abrams, Robert Zack [using keystroke data]
Other – Karina Hernandez, Mark Ritzmann [using keystroke data], (John Galatti)