1. Pattern Recognition: Statistical and Neural
Nanjing University of Science & Technology
Pattern Recognition: Statistical and Neural
Lonnie C. Ludeman
Lecture 17
Oct 21, 2005

2. Lecture 17 Topics View of Perceptron Algorithm in Pattern Space
Fractional Correction Perceptron Algorithm
Simplified Perceptron Algorithm
4. Derivation of the perceptron algorithm
5. Extension of Perceptron Algorithm to M Class Case: 3 Special Cases

3. from C1
from C2
Motivation

4. How do we find separating Hyperplane??? a “needle in the haystack”
Question:
How do we find separating Hyperplane??? a “needle in the haystack”
Answer:
The Perceptron Algorithm !!!!
Other ways exist like random selection

5. from C1
from C2
Motivation
Separating Hyperplane

6. Linear Discriminant Functions
where
Augmented Pattern Vector
Weight vector C1
Decision Rule:
if d(x) > 0
< C2

7. Finds a hyperplane that separates two sets of patterns
Review Perceptron Algorithm
Finds a hyperplane that separates two sets of patterns
Algorithm

8. Review Perceptron Algorithm
New Training Sample

9. Review Perceptron Algorithm
wT(k)x(k) < 0
wT(k)x(k) > 0
If c is too large we may not get convergence

10. View of Perceptron correction in pattern space

11. Fractional Correction Perceptron Algorithm
Weight Update Equation
Where

12. Weight update in original pattern space

13. Weight update in original pattern space

14. Weight update in original pattern space

15. Simplified Perceptron Algorithm
Given the following samples from two classes
We want a weight vector w such that
Negated patterns for class C2

16. We could define two classes as
Consider the set of samples x of C11 U C21
then the weight vector update is as follows
That is all of the samples should be on the positive side of the hyperplane boundary

17. This is a simplification of the perceptron algorithm in that it only contains one if branch
Simplified algorithm:
1. Augment all patterns in C1 and C2
2. Negate the augmented patterns of C2
3. Combine the two sets above to form one set of samples
4. Iterate through this set using the weight update as

18. Derivation of the Perceptron Algorithm
Define the following performance measure

19. Minimizing J for each sample will satisfy the conditions desired
To minimize J we can use the gradient algorithm where the weight update is as follows

20. The partial derivatives with respect to each weight are determined as

21. Substituting these partials into the weight update equation yields the following
for

22. Which can be written in the following vector form
Which is the perceptron algorithm in augmented and negated form.
(end of proof)

23. Other Perceptron like Algorithms
If we use different performance measures in the preceding proof we get perceptron like algorithms to accomplish the separation of classes.

24. Other meaningful performance measures
Each leads to different weight update equations. Each requires a convergence theorem to be useful.

25. So far only worked with separating two classes
So far only worked with separating two classes. Most problems have more than two classes.
Question:
Can we modify the perceptron algorithm to work with more than two classes?
Answer:
Yes for certain special cases.

26. Case 1. Pattern classes Group separable

27. Case 1. K Pattern classes - Group separable
Given:
S1 is set of samples from class C1 S2 is set of samples from class C2
SK is set of samples from class CK
S = S1 U S2 U U SK …
Define:
Assume: Pattern classes are group separable
S1 linearly separable from S1/ = S – S1 S2 linearly separable from S2/ = S – S2
SK linearly separable from SK/ = S – SK …

28. dk(x)=wk1x1 + wk2x2 + … + wknxn + wkn+1
Find K Linear Discriminant functions that separate the Sk in the following manner
Discriminant Functions
for k = 1, 2, … , K
dk(x)=wk1x1 + wk2x2 + … + wknxn + wkn+1
Decision Rule: If dj(x) > 0 then decide Cj
Solution:
Find dk(x) by using the Perceptron algorithm on the two classes Sk and Sk/

29. Solution Continued
(1) Find d1(x) by using the Perceptron algorithm on the two sets S1 and S1/
(2) Find d2(x) by using the Perceptron algorithm on the two sets S2 and S2/ …
(K) Find dK(x) by using the Perceptron algorithm on the two sets SK and SK/

30. 0 = wk1x1 + wk2x2 + … + wknxn + wkn+1
Solution Continued
This gives the decision boundaries as:
for k = 1, 2, … , K
0 = wk1x1 + wk2x2 + … + wknxn + wkn+1

31. Case 2. Pattern classes Pairwise separable

32. dkj(x)=wkj1x1 + wkj2x2 + … + wkjnxn + wkjn+1
Case 2. Pattern classes Pairwise separable
Find Linear Discriminant functions that separate all pairs of Sk in the following manner:
for k = 1, 2, … , K
j = 1, 2, … , K j = k
dkj(x)=wkj1x1 + wkj2x2 + … + wkjnxn + wkjn+1
Decision Rule:
for k = 1, 2, … , K
If dkj(x) > 0 for all j , j = k then decide Ck
On Boundaries decide randomly

33. Solution:
for k = 1, 2, … , K
j = 1, 2, … , K , j = k
Find dkj(x) by using the Perceptron algorithm on the two classes Sk and Sj .
Notes:
(1) dkj(x) = - djk(x) .
(2) Requires determining K(K-1)/2 discriminant functions

34. Case 3. K Pattern classes separable by K discriminant functions

37. Flow Diagram for Case 3 Perceptron separability

38. Summary Lecture 17 View of Perceptron Algorithm in Pattern Space
Fractional Correction Perceptron Algorithm
Simplified Perceptron Algorithm
4. Derivation of the perceptron algorithm
5. Extension of Perceptron Algorithm to M Class Case: 3 Special Cases

39. End of Lecture 17