FUZZ-IEEE 2003 1 Kernel Machines and Additive Fuzzy Systems: Classification and Function Approximation Yixin Chen and James Z. Wang The Pennsylvania State.

Slides:

Advertisements

Similar presentations

Statistical Machine Learning- The Basic Approach and Current Research Challenges Shai Ben-David CS497 February, 2007.

Advertisements

Introduction to Support Vector Machines (SVM)

Support Vector Machine

Lecture 9 Support Vector Machines

ECG Signal processing (2)

Image classification Given the bag-of-features representations of images from different classes, how do we learn a model for distinguishing them?

Notes Sample vs distribution “m” vs “µ” and “s” vs “σ” Bias/Variance Bias: Measures how much the learnt model is wrong disregarding noise Variance: Measures.

SVM - Support Vector Machines A new classification method for both linear and nonlinear data It uses a nonlinear mapping to transform the original training.

Classification / Regression Support Vector Machines

Pattern Recognition and Machine Learning

An Introduction of Support Vector Machine

Support Vector Machines and Kernels Adapted from slides by Tim Oates Cognition, Robotics, and Learning (CORAL) Lab University of Maryland Baltimore County.

Support Vector Machines

SVM—Support Vector Machines

Model Assessment and Selection

Second order cone programming approaches for handing missing and uncertain data P. K. Shivaswamy, C. Bhattacharyya and A. J. Smola Discussion led by Qi.

Support Vector Machines (and Kernel Methods in general)

Fuzzy Support Vector Machines (FSVMs) Weijia Wang, Huanren Zhang, Vijendra Purohit, Aditi Gupta.

Support Vector Machines (SVMs) Chapter 5 (Duda et al.)

The Nature of Statistical Learning Theory by V. Vapnik

University of Texas at Austin Machine Learning Group Department of Computer Sciences University of Texas at Austin Support Vector Machines.

Variations of Minimax Probability Machine Huang, Kaizhu

Support Vector Machines Based on Burges (1998), Scholkopf (1998), Cristianini and Shawe-Taylor (2000), and Hastie et al. (2001) David Madigan.

Binary Classification Problem Learn a Classifier from the Training Set

Sparse Kernels Methods Steve Gunn.

1 Computational Learning Theory and Kernel Methods Tianyi Jiang March 8, 2004.

SVM Support Vectors Machines

What is Learning All about ?  Get knowledge of by study, experience, or being taught  Become aware by information or from observation  Commit to memory.

Greg GrudicIntro AI1 Support Vector Machine (SVM) Classification Greg Grudic.

Computational Learning Theory

Statistical Learning Theory: Classification Using Support Vector Machines John DiMona Some slides based on Prof Andrew Moore at CMU:

Optimization Theory Primal Optimization Problem subject to: Primal Optimal Value:

Machine Learning Week 4 Lecture 1. Hand In Data Is coming online later today. I keep test set with approx test images That will be your real test.

CS 8751 ML & KDDSupport Vector Machines1 Support Vector Machines (SVMs) Learning mechanism based on linear programming Chooses a separating plane based.

Support Vector Machines Mei-Chen Yeh 04/20/2010. The Classification Problem Label instances, usually represented by feature vectors, into one of the predefined.

1 SUPPORT VECTOR MACHINES İsmail GÜNEŞ. 2 What is SVM? A new generation learning system. A new generation learning system. Based on recent advances in.

An Introduction to Support Vector Machine (SVM) Presenter : Ahey Date : 2007/07/20 The slides are based on lecture notes of Prof. 林智仁 and Daniel Yeung.

Overview of Supervised Learning Overview of Supervised Learning2 Outline Linear Regression and Nearest Neighbors method Statistical Decision.

Support Vector Machines Reading: Ben-Hur and Weston, “A User’s Guide to Support Vector Machines” (linked from class web page)

CS Statistical Machine learning Lecture 18 Yuan (Alan) Qi Purdue CS Oct

1 Chapter 6. Classification and Prediction Overview Classification algorithms and methods Decision tree induction Bayesian classification Lazy learning.

Classification Derek Hoiem CS 598, Spring 2009 Jan 27, 2009.

START OF DAY 5 Reading: Chap. 8. Support Vector Machine.

Ohad Hageby IDC Support Vector Machines & Kernel Machines IP Seminar 2008 IDC Herzliya.

1 Support Vector Machine (SVM)  MUMT611  Beinan Li  Music McGill 

Support Vector Machines Tao Department of computer science University of Illinois.

Chapter1: Introduction Chapter2: Overview of Supervised Learning

Chapter 20 Classification and Estimation Classification – Feature selection Good feature have four characteristics: –Discrimination. Features.

Text Categorization With Support Vector Machines: Learning With Many Relevant Features By Thornsten Joachims Presented By Meghneel Gore.

Final Exam Review CS479/679 Pattern Recognition Dr. George Bebis 1.

Goal of Learning Algorithms  The early learning algorithms were designed to find such an accurate fit to the data.  A classifier is said to be consistent.

Support Vector Machines (SVM): A Tool for Machine Learning Yixin Chen Ph.D Candidate, CSE 1/10/2002.

Feature Selction for SVMs J. Weston et al., NIPS 2000 오장민 (2000/01/04) Second reference : Mark A. Holl, Correlation-based Feature Selection for Machine.

Support Vector Machines Reading: Ben-Hur and Weston, “A User’s Guide to Support Vector Machines” (linked from class web page)

Greg GrudicIntro AI1 Support Vector Machine (SVM) Classification Greg Grudic.

CSSE463: Image Recognition Day 14 Lab due Weds. Lab due Weds. These solutions assume that you don't threshold the shapes.ppt image: Shape1: elongation.

Generalization Error of pac Model  Let be a set of training examples chosen i.i.d. according to  Treat the generalization error as a r.v. depending on.

High resolution product by SVM. L’Aquila experience and prospects for the validation site R. Anniballe DIET- Sapienza University of Rome.

Support Vector Machines (SVMs) Chapter 5 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis.

Support Vector Machines

CS 9633 Machine Learning Support Vector Machines

Project 4: Facial Image Analysis with Support Vector Machines

An Introduction to Support Vector Machines

Pawan Lingras and Cory Butz

Support Vector Machines Introduction to Data Mining, 2nd Edition by

Computational Learning Theory

CSSE463: Image Recognition Day 14

Computational Learning Theory

Support Vector Machines 2

Presentation transcript:

FUZZ-IEEE Kernel Machines and Additive Fuzzy Systems: Classification and Function Approximation Yixin Chen and James Z. Wang The Pennsylvania State University

FUZZ-IEEE Outline Introduction VC theory and Support Vector Machines Additive fuzzy systems and kernel machines Support vector learning for a class of additive fuzzy systems Experimental results Conclusions and future work

FUZZ-IEEE Introduction Building a fuzzy system Structure identification Parameter estimation Model validation Do we get a good fuzzy model? How capable can a fuzzy model be? How well can the model generalize?

FUZZ-IEEE Introduction (continued) Several types of fuzzy models are “universal approximators” Generalization performance Structural risk minimization Bias variance dilemma Overfitting phenomena  A “right” tradeoff between training accuracy and model complexity

FUZZ-IEEE Introduction (continued) Two approaches to find a “right” tradeoff Cross-validation for model selection Model reduction to simplify the model Vapnik-Chervonenkis (VC) theory A general measure of model set complexity Bounds on generalization Support Vector Machines (SVM)

FUZZ-IEEE Outline Introduction VC theory and Support Vector Machines Additive fuzzy systems and kernel machines Support vector learning for a class of additive fuzzy systems Experimental results Conclusions and future work

FUZZ-IEEE VC Theory and Support Vector Machines One result from VC Theory Binary classification: given a set of training samples drawn independently from some unknown distribution, with probability, the probability of misclassification for any decision function is bounded above by

FUZZ-IEEE VC Theory and Support Vector Machines (continued) Support Vector Machines (SVMs) Optimal separating hyperplane

FUZZ-IEEE VC Theory and Support Vector Machines (continued) Kernel trick A Mercer kernel is a function,, satisfying where is sometimes referred to as the Mercer features

FUZZ-IEEE VC Theory and Support Vector Machines (continued) Quadratic programming Decision function

FUZZ-IEEE Outline Introduction VC theory and Support Vector Machines Additive fuzzy systems and kernel machines Support vector learning for a class of additive fuzzy systems Experimental results Conclusions and future work

FUZZ-IEEE Additive Fuzzy Systems and Kernel Machines Kernel Machines A class of additive fuzzy systems is functionally equivalent to a class of kernel machines

FUZZ-IEEE Additive Fuzzy Systems and Kernel Machines (continued) Additive Fuzzy System (AFS) m fuzzy rules of the form Product as fuzzy conjunction operator Addition for fuzzy rule aggregation First order moment defuzzification Reference function Kernel is the product of reference functions

FUZZ-IEEE Additive Fuzzy Systems and Kernel Machines (continued) Positive definite fuzzy systems (PDFS) Reference functions are positive definite functions  Mercer kernels Examples: GaussianSymmetric triangle CauchyHyperbolic secant LaplaceSquared sinc

FUZZ-IEEE Outline Introduction VC theory and Support Vector Machines Additive fuzzy systems and kernel machines Support vector learning for a class of additive fuzzy systems Experimental results Conclusions and future work

FUZZ-IEEE Support Vector Learning for a Class of Additive Fuzzy Systems SVMPDFS KernelReference functions Support vectorsIF-part of fuzzy rules Lagrange multiplierTHEN-part of fuzzy rules

FUZZ-IEEE Outline Introduction VC theory and Support Vector Machines Additive fuzzy systems and kernel machines Support vector learning for a class of additive fuzzy systems Experimental results Conclusions and future work

FUZZ-IEEE Experimental Results USPS data set Training data (7291), testing data (2007) 5-fold cross- validation to determine parameters

FUZZ-IEEE Experimental Results (continued) Linear SVM:91.3% k-nearest neighbor:94.3%

FUZZ-IEEE Outline Introduction VC theory and Support Vector Machines Additive fuzzy systems and kernel machines Support vector learning for a class of additive fuzzy systems Experimental results Conclusions and future work

FUZZ-IEEE Conclusions and Future Work Kernel Machines Fuzzy Systems PDFS