Quadratic Perceptron Learning with Applications Tonghua Su National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences Beijing, PR China Dec 2, 2010

Outline Introduction Motivations Quadratic Perceptron Algorithm Previous works Theory perspective Practical perspective Open issues Conclusions

1 Introduction Notation, binary classification, multi-class classification, large scale learning vs large category learning

Introduction Domain Set Label Set Training Data Binary Classification e.g. linear model

Introduction Multi-class Classification Learning strategy One vs one One vs all Single machine e.g. Linear model Large-category classification Chinese character recognition (3,755 classes) More confusions between classes

Introduction Large Scale Learning Large Category vs Large Scale large numbers of data points high dimensions Challenge in computation resource Large Category vs Large Scale Almost certainly: large category  large scale Tradeoffs: efficiency vs accuracy

2 Motivations MQDF HMMs

Modified Quadratic Discriminant Function (MQDF) MQDF [Kimura et al ‘1987] Using SVD, Truncate small eigenvalues

Modified Quadratic Discriminant Function (MQDF) MQDF+MCE+Synthetic samples [Chen et al ‘2010] Building block: discriminative learning of MQDF

Hidden Markov Models (HMMs) Markovian transition + state specific generator Continuous density HMMs: each state emits a GMM e.g. Usable in handwritten Chinese text recognition [Su ‘2007] F L 0.05 0.95 F L S E

Hidden Markov Models (HMMs) Perceptron training of HMMs [Cheng et al ’2009] Joint distribution Discriminant function log p(s,x) Perceptron training Nonnegative-definite constraint Lack of theoretical foundation ‘

3 Quadratic Perceptron Algorithm Related works Theoretical considerations Practical considerations Open issues

Previous Works Rosenblatt’s Perceptron [Rosenblatt ’58] Updating rule:

Previous Works Rosenblatt’s Perceptron w0 wTx3y3=0 w2 _ + wTx2y2=0 Solution Region x2y2 w1 wTx4y4=0 _ x3y3 w2 + x2y2 w3 + _ x3y3 w4 wTx1y1=0 + _

Previous Works Rosenblatt’s Perceptron [Rosenblatt ’58] View from batch loss where Using stochastic gradient decent (SGD) 立陶宛

Previous Works Convergence Theorem [Block ’62,Novikoff ’62] Linearly separate data Stop at most (R/)2 steps

Previous Works Voted Perceptron [Freund ’99] Training algorithm Prediction:

Previous Works Voted Perceptron Generalization bound

Previous Works Perceptron with Margin [Krauth ’87, Li ’2002]

Previous Works Ballseptron [Shalev-Shwartz ’2005]

Previous Works Perceptron with Unlearning [Panagiotakopoulos ’2010]

Theoretical Perspective Prediction rule Learning 立陶宛 Lithuanian Dataset [,lɪθju'enɪə ]

Theoretical Perspective Algorithm online version

Theoretical Perspective Convergence Theorem of Quadratic Perceptron (quadratic separable)

Theoretical Perspective Convergence Theorem of Quadratic Perceptron with Magin (quadratic separable)

Theoretical Perspective Bounds for quadratic inseparable case

Theoretical Perspective Generalization Bound

Theoretical Perspective Nonnegative-definite constraints Projection to the valid space Restriction on updating Convergence holds

Theoretical Perspective Toy problem: Lithuanian Dataset 4000 training instances 2000 test instances

Theoretical Perspective Perceptron learning (toy problem)

Theoretical Perspective Extension to Multi-class QDF

Theoretical Perspective Extension to Multi-class QDF Theoretical property holds as binary QDF Proof can be completed using Kesler’s construction

Practical Perspective Perceptron batch loss where SGD

Practical Perspective Constant margin Dynamic margin

Practical Perspective Experiments Benchmark on digit databases

Practical Perspective Experiments Benchmark on digit databases grg on MNIST

Practical Perspective Experiments Benchmark on digit databases grg on USPS

Practical Perspective Experiments Effects of training size (grg on MNIST)

Practical Perspective Experiments Benchmark on CASIA-HWDB1.1

Practical Perspective Experiments Benchmark on CASIA-HWDB1.1

Open Issues Convergence on GMM/MQDF? Error reduction on CASIA-DB1.1 is small How about adding more data ? Can label permutation help? Speedup the training process Evaluate on more datasets

4 Conclusions

Conclusions Theoretical foundation for QDF Perceptron learning of MQDF Convergence Theorem Generalization Bound Perceptron learning of MQDF Margin is need for good generalization More data may help

Thank you!

References [Chen et al ‘2010] Xia Chen, Tong-Hua Su,Tian-Wen Zhang. Discriminative Training of MQDF Classifier on Synthetic Chinese String Samples, CCPR,2010 [Cheng et al ‘2009] C. Cheng, F. Sha, L. Saul. Matrix updates for perceptron training of continuous density hidden markov models, ICML, 2009. [Kimura ‘87] F. Kimura, K. Takashina, S. Tsuruoka, Y. Miyake. Modified quadratic discriminant functions and the application to Chinese character recognition, IEEE TPAMI, 9(1): 149-153, 1987. [Panagiotakopoulos ‘2010] C. Panagiotakopoulos, P. Tsampouka. The Margin Perceptron with Unlearning, ICML, 2010. [Krauth ‘87] W. Krauth and M. Mezard. Learning algorithms with optimal stability in neural networks. Journal of Physics A, 20, 745-752, 1987. [Li ‘2002] Yaoyong Li, Hugo Zaragoza, Ralf Herbrich, John Shawe-Taylor, Jaz Kandola. The Perceptron Algorithm with Uneven Margins, ICML, 2002.

References [Freund ‘99] Y. Freund and R. E. Schapire. Large margin classification using the perceptron algorithm. Machine Learning, 37(3): 277-296, 1999. [Shalev-Shwartz ’2005] Shai Shalev-Shwartz, Yoram Singer. A New Perspective on an Old Perceptron Algorithm, COLT, 2005. [Novikoff ‘62] A. B. J. Novikoff. On convergence proofs on perceptrons. In Proc. Symp. Math. Theory Automata, Vol.12, pp. 615–622, 1962. [Rosenblatt ‘58] Rosenblatt, F. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65 (6):386–408, 1958. [Block ‘62] H.D. Block. The perceptron: A model for brain functioning, Reviews of Modern Phsics, 1962, 34:123-135.