1. FUZZ-IEEE 2003 1 Kernel Machines and Additive Fuzzy Systems: Classification and Function Approximation Yixin Chen and James Z. Wang The Pennsylvania State University http://www.cse.psu.edu/~yixchen

2. FUZZ-IEEE 20032 Outline Introduction VC theory and Support Vector Machines Additive fuzzy systems and kernel machines Support vector learning for a class of additive fuzzy systems Experimental results Conclusions and future work

3. FUZZ-IEEE 20033 Introduction Building a fuzzy system Structure identification Parameter estimation Model validation Do we get a good fuzzy model? How capable can a fuzzy model be? How well can the model generalize?

4. FUZZ-IEEE 20034 Introduction (continued) Several types of fuzzy models are “universal approximators” Generalization performance Structural risk minimization Bias variance dilemma Overfitting phenomena  A “right” tradeoff between training accuracy and model complexity

5. FUZZ-IEEE 20035 Introduction (continued) Two approaches to find a “right” tradeoff Cross-validation for model selection Model reduction to simplify the model Vapnik-Chervonenkis (VC) theory A general measure of model set complexity Bounds on generalization Support Vector Machines (SVM)

6. FUZZ-IEEE 20036 Outline Introduction VC theory and Support Vector Machines Additive fuzzy systems and kernel machines Support vector learning for a class of additive fuzzy systems Experimental results Conclusions and future work

7. FUZZ-IEEE 20037 VC Theory and Support Vector Machines One result from VC Theory Binary classification: given a set of training samples drawn independently from some unknown distribution, with probability, the probability of misclassification for any decision function is bounded above by

8. FUZZ-IEEE 20038 VC Theory and Support Vector Machines (continued) Support Vector Machines (SVMs) Optimal separating hyperplane

9. FUZZ-IEEE 20039 VC Theory and Support Vector Machines (continued) Kernel trick A Mercer kernel is a function,, satisfying where is sometimes referred to as the Mercer features

10. FUZZ-IEEE 200310 VC Theory and Support Vector Machines (continued) Quadratic programming Decision function

11. FUZZ-IEEE 200311 Outline Introduction VC theory and Support Vector Machines Additive fuzzy systems and kernel machines Support vector learning for a class of additive fuzzy systems Experimental results Conclusions and future work

12. FUZZ-IEEE 200312 Additive Fuzzy Systems and Kernel Machines Kernel Machines A class of additive fuzzy systems is functionally equivalent to a class of kernel machines

13. FUZZ-IEEE 200313 Additive Fuzzy Systems and Kernel Machines (continued) Additive Fuzzy System (AFS) m fuzzy rules of the form Product as fuzzy conjunction operator Addition for fuzzy rule aggregation First order moment defuzzification Reference function Kernel is the product of reference functions

14. FUZZ-IEEE 200314 Additive Fuzzy Systems and Kernel Machines (continued) Positive definite fuzzy systems (PDFS) Reference functions are positive definite functions  Mercer kernels Examples: GaussianSymmetric triangle CauchyHyperbolic secant LaplaceSquared sinc

15. FUZZ-IEEE 200315 Outline Introduction VC theory and Support Vector Machines Additive fuzzy systems and kernel machines Support vector learning for a class of additive fuzzy systems Experimental results Conclusions and future work

16. FUZZ-IEEE 200316 Support Vector Learning for a Class of Additive Fuzzy Systems SVMPDFS KernelReference functions Support vectorsIF-part of fuzzy rules Lagrange multiplierTHEN-part of fuzzy rules

17. FUZZ-IEEE 200317 Outline Introduction VC theory and Support Vector Machines Additive fuzzy systems and kernel machines Support vector learning for a class of additive fuzzy systems Experimental results Conclusions and future work

18. FUZZ-IEEE 200318 Experimental Results USPS data set Training data (7291), testing data (2007) 5-fold cross- validation to determine parameters

19. FUZZ-IEEE 200319 Experimental Results (continued) Linear SVM:91.3% k-nearest neighbor:94.3%

20. FUZZ-IEEE 200320 Outline Introduction VC theory and Support Vector Machines Additive fuzzy systems and kernel machines Support vector learning for a class of additive fuzzy systems Experimental results Conclusions and future work

21. FUZZ-IEEE 200321 Conclusions and Future Work Kernel Machines Fuzzy Systems PDFS