An Introduction of Support Vector Machine

Slides:



Advertisements
Similar presentations
Support Vector Machine & Its Applications
Advertisements

Introduction to Support Vector Machines (SVM)
Support Vector Machines
Lecture 9 Support Vector Machines
ECG Signal processing (2)
Image classification Given the bag-of-features representations of images from different classes, how do we learn a model for distinguishing them?
Support Vector Machine & Its Applications Abhishek Sharma Dept. of EEE BIT Mesra Aug 16, 2010 Course: Neural Network Professor: Dr. B.M. Karan Semester.
S UPPORT V ECTOR M ACHINES Jianping Fan Dept of Computer Science UNC-Charlotte.
Support Vector Machine & Its Applications Mingyue Tan The University of British Columbia Nov 26, 2004 A portion (1/3) of the slides are taken from Prof.
SVM - Support Vector Machines A new classification method for both linear and nonlinear data It uses a nonlinear mapping to transform the original training.
ONLINE ARABIC HANDWRITING RECOGNITION By George Kour Supervised by Dr. Raid Saabne.
Classification / Regression Support Vector Machines
An Introduction of Support Vector Machine
Support Vector Machines
SVM—Support Vector Machines
Machine learning continued Image source:
LPP-HOG: A New Local Image Descriptor for Fast Human Detection Andy Qing Jun Wang and Ru Bo Zhang IEEE International Symposium.
Support Vector Machines
Image classification Given the bag-of-features representations of images from different classes, how do we learn a model for distinguishing them?
Support Vector Machines (SVMs) Chapter 5 (Duda et al.)
University of Texas at Austin Machine Learning Group Department of Computer Sciences University of Texas at Austin Support Vector Machines.
1 Classification: Definition Given a collection of records (training set ) Each record contains a set of attributes, one of the attributes is the class.
Support Vector Classification (Linearly Separable Case, Primal) The hyperplanethat solves the minimization problem: realizes the maximal margin hyperplane.
Classification Problem 2-Category Linearly Separable Case A- A+ Malignant Benign.
Support Vector Machines
CS 4700: Foundations of Artificial Intelligence
Support Vector Machines
Greg GrudicIntro AI1 Support Vector Machine (SVM) Classification Greg Grudic.
An Introduction to Support Vector Machines Martin Law.
Classification III Tamara Berg CS Artificial Intelligence Many slides throughout the course adapted from Svetlana Lazebnik, Dan Klein, Stuart Russell,
Support Vector Machine & Image Classification Applications
CS 8751 ML & KDDSupport Vector Machines1 Support Vector Machines (SVMs) Learning mechanism based on linear programming Chooses a separating plane based.
Support Vector Machines Mei-Chen Yeh 04/20/2010. The Classification Problem Label instances, usually represented by feature vectors, into one of the predefined.
1 SUPPORT VECTOR MACHINES İsmail GÜNEŞ. 2 What is SVM? A new generation learning system. A new generation learning system. Based on recent advances in.
计算机学院 计算感知 Support Vector Machines. 2 University of Texas at Austin Machine Learning Group 计算感知 计算机学院 Perceptron Revisited: Linear Separators Binary classification.
An Introduction to Support Vector Machine (SVM) Presenter : Ahey Date : 2007/07/20 The slides are based on lecture notes of Prof. 林智仁 and Daniel Yeung.
Kernel Methods A B M Shawkat Ali 1 2 Data Mining ¤ DM or KDD (Knowledge Discovery in Databases) Extracting previously unknown, valid, and actionable.
SVM Support Vector Machines Presented by: Anas Assiri Supervisor Prof. Dr. Mohamed Batouche.
Classifiers Given a feature representation for images, how do we learn a model for distinguishing features from different classes? Zebra Non-zebra Decision.
An Introduction to Support Vector Machines (M. Law)
CISC667, F05, Lec22, Liao1 CISC 667 Intro to Bioinformatics (Fall 2005) Support Vector Machines I.
CS 478 – Tools for Machine Learning and Data Mining SVM.
An Introduction to Support Vector Machine (SVM)
University of Texas at Austin Machine Learning Group Department of Computer Sciences University of Texas at Austin Support Vector Machines.
Support vector machine LING 572 Fei Xia Week 8: 2/23/2010 TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A 1.
Support Vector Machines. Notation Assume a binary classification problem. –Instances are represented by vector x   n. –Training examples: x = (x 1,
Text Classification using Support Vector Machine Debapriyo Majumdar Information Retrieval – Spring 2015 Indian Statistical Institute Kolkata.
Support Vector Machines Reading: Ben-Hur and Weston, “A User’s Guide to Support Vector Machines” (linked from class web page)
Greg GrudicIntro AI1 Support Vector Machine (SVM) Classification Greg Grudic.
An Introduction of Support Vector Machine In part from of Jinwei Gu.
1 Kernel Machines A relatively new learning methodology (1992) derived from statistical learning theory. Became famous when it gave accuracy comparable.
Roughly overview of Support vector machines Reference: 1.Support vector machines and machine learning on documents. Christopher D. Manning, Prabhakar Raghavan.
A Brief Introduction to Support Vector Machine (SVM) Most slides were from Prof. A. W. Moore, School of Computer Science, Carnegie Mellon University.
An Introduction of Support Vector Machine Courtesy of Jinwei Gu.
Support Vector Machine Slides from Andrew Moore and Mingyue Tan.
PREDICT 422: Practical Machine Learning
Support Vector Machines
Support Vector Machines
An Introduction to Support Vector Machines
Support Vector Machines
Statistical Learning Dong Liu Dept. EEIS, USTC.
CS 2750: Machine Learning Support Vector Machines
CSSE463: Image Recognition Day 14
Support Vector Machine
Support Vector Machines
Introduction to Support Vector Machines
CSSE463: Image Recognition Day 14
Support Vector Machines
Support vector machines
SVMs for Document Ranking
Presentation transcript:

An Introduction of Support Vector Machine Courtesy of Jinwei Gu

Today: Support Vector Machine (SVM) A classifier derived from statistical learning theory by Vapnik, et al. in 1992 SVM became famous when, using images as input, it gave accuracy comparable to neural-network with hand-designed features in a handwriting recognition task Currently, SVM is widely used in object detection & recognition, content-based image retrieval, text recognition, biometrics, speech recognition, etc.

Discriminant Function It can be arbitrary functions of x, such as: Decision Tree Linear Functions Nonlinear Functions

Linear Discriminant Function g(x) is a linear function: x2 wT x + b > 0 g(x) is a hyper-plane in the n-dimensional feature space, w and x vectors w1x1+w2x2+b=0 wT x + b = 0 n (Unit-length) normal vector of the hyper-plane: wT x + b < 0 x1

Linear Discriminant Function denotes +1 denotes -1 How would you classify these points using a linear discriminant function in order to minimize the error rate? x2 Infinite number of answers! x1

Linear Discriminant Function denotes +1 denotes -1 How would you classify these points using a linear discriminant function in order to minimize the error rate? x2 Infinite number of answers! x1

Linear Discriminant Function denotes +1 denotes -1 How would you classify these points using a linear discriminant function in order to minimize the error rate? x2 Infinite number of answers! x1

Linear Discriminant Function denotes +1 denotes -1 How would you classify these points using a linear discriminant function in order to minimize the error rate? x2 Infinite number of answers! Which one is the best? x1

Large Margin Linear Classifier denotes +1 denotes -1 The linear discriminant function (classifier) with the maximum margin is the best x2 Margin “safe zone” Margin is defined as the width that the boundary could be increased by before hitting a data point Why it is the best? Robust to outliners and thus strong generalization ability x1

Large Margin Linear Classifier denotes +1 denotes -1 Given a set of data points: x2 , where Note: class is +1,-1 NOT 1,0) With a scale transformation on both w and b, the above is equivalent to x1

Large Margin Linear Classifier denotes +1 denotes -1 We know that x2 Margin x+ x- wT x + b = 1 Support Vectors wT x + b = 0 wT x + b = -1 The margin width is: n x1

Large Margin Linear Classifier denotes +1 denotes -1 Formulation: x2 Margin x+ x- wT x + b = 1 such that wT x + b = 0 wT x + b = -1 n x1 Remember: our objective is finding w, b such that margins are maximized

Large Margin Linear Classifier denotes +1 denotes -1 Equivalent formulation: x2 Margin x+ x- wT x + b = 1 such that wT x + b = 0 wT x + b = -1 n x1

Large Margin Linear Classifier denotes +1 denotes -1 Equivalent formulation: x2 Margin x+ x- wT x + b = 1 such that wT x + b = 0 wT x + b = -1 n x1

Solving the Optimization Problem s.t. Quadratic programming with linear constraints s.t. Lagrangian Function

Lagrangian Given a function f(x) and a set of constraints c1..cn, a Lagrangian is a function L(f,c1,..cn,1,.. n) that “incorporates” constraints in the optimization problem The optimum is at a point where (Karush-Kuhn-Tucker (KKT) conditions): The third condition is known as complementarity condition

Solving the Optimization Problem s.t. To minimize L we need to maximize the red box

Dual Problem Formulation (2) s.t. Since and Because of complementary condition 3 Lagrangian Dual Problem s.t. , and Each non-zero αi indicates that corresponding xi is a support vector SV.

Why only SV? The complementarity condition (the third one of KKT, see previous definition of Lagrangians) implies that one of the 2 multiplicands is zero. However, for non-SV xi. Therefore only data points on the margin (=SV) will have a non-zero α (because they are the only ones for wich yi (wTxi + b ) = 1 )

Final Formulation To summaryze: Again, remember that the constraint (2) can be verified with a non-equality only for the SV!! Find α1…αn such that Q(α) =Σαi - ½ΣΣαiαjyiyjxiTxj is maximized and (1) Σαiyi = 0 (2) αi ≥ 0 for all αi

The Optimization Problem Solution Given a solution α1…αn to the dual problem Q(α) , solution to the primal is: Each non-zero αi indicates that corresponding xi is a support vector. Then the classifying function for a new point x is (note that we don’t need w explicitly): Notice that it relies on an inner product xiTx between the test point x and the support vectors xi . Only SV are useful for classification of new instances!!. Also keep in mind that solving the optimization problem involved computing the inner products xiTxj between all training points. w =Σαiyixi b = yk - Σαiyixi Txk for any αk > 0 g(x) = ΣαiyixiTx + b (xi are SV)

Working example A 3-point training set: <(1,1)+><(2,0)+><(2,3),-> The maximum margin weight vector will be parallel to the shortest line connecting points of the two classes, that is, the line between (1,1) and (2,3) (the 2 Support Vectors) W

Margin computation wTx1+b=+1 wTx2+b=-1

Soft Margin Classification What if the training set is not linearly separable? Slack variables ξi can be added to allow misclassification of difficult or noisy examples, resulting margins are called soft. ξi ξi xi are the misclassified examples

Large Margin Linear Classifier Formulation: such that Parameter C can be viewed as a way to control over-fitting.

Large Margin Linear Classifier Formulation: (Lagrangian Dual Problem) such that

Non-linear SVMs Datasets that are linearly separable with noise work out great: x x But what are we going to do if the dataset is just too hard? How about… mapping data to a higher-dimensional space: x x2 This slide is courtesy of www.iro.umontreal.ca/~pift6080/documents/papers/svm_tutorial.ppt

Non-linear SVMs: Feature Space General idea: the original input space can be mapped to some higher-dimensional feature space where the training set is separable: Φ: x → φ(x) This slide is courtesy of www.iro.umontreal.ca/~pift6080/documents/papers/svm_tutorial.ppt

The original objects (left side of the schematic) are mapped, i. e The original objects (left side of the schematic) are mapped, i.e., rearranged, using a set of mathematical functions, known as kernels. φ(x)

Nonlinear SVMs: The Kernel Trick With this mapping, our discriminant function is now: Original formulation was g(x) = ΣαiyixiTx + b No need to know this mapping explicitly, because we only use the dot product of feature vectors in both the training and test. A kernel function is defined as a function that corresponds to a dot product of two feature vectors in some expanded feature space:

Nonlinear SVMs: The Kernel Trick An example: 2-dimensional vectors x=[x1 x2]; let K(xi,xj)=(1 + xiTxj)2, Need to show that K(xi,xj) = φ(xi) Tφ(xj): K(xi,xj)=(1 + xiTxj)2, = 1+ xi12xj12 + 2 xi1xj1 xi2xj2+ xi22xj22 + 2xi1xj1 + 2xi2xj2 = [1 xi12 √2 xi1xi2 xi22 √2xi1 √2xi2]T [1 xj12 √2 xj1xj2 xj22 √2xj1 √2xj2] = φ(xi) Tφ(xj), where φ(x) = [1 x12 √2 x1x2 x22 √2x1 √2x2] This slide is courtesy of www.iro.umontreal.ca/~pift6080/documents/papers/svm_tutorial.ppt

Nonlinear SVMs: The Kernel Trick Examples of commonly-used kernel functions: Linear kernel: Polynomial kernel: Gaussian (Radial-Basis Function (RBF) ) kernel: Sigmoid: In general, functions that satisfy Mercer’s condition can be kernel functions.

Nonlinear SVM: Optimization Formulation: (Lagrangian Dual Problem) such that The solution of the discriminant function is The optimization technique is the same.

Support Vector Machine: Algorithm 1. Choose a kernel function 2. Choose a value for C 3. Solve the quadratic programming problem (many software packages available) 4. Construct the discriminant function from the support vectors

Some Issues Choice of kernel Choice of kernel parameters - Gaussian or polynomial kernel is default - if ineffective, more elaborate kernels are needed - domain experts can give assistance in formulating appropriate similarity measures Choice of kernel parameters - e.g. σ in Gaussian kernel - σ is the distance between closest points with different classifications - In the absence of reliable criteria, applications rely on the use of a validation set or cross-validation to set such parameters. Optimization criterion – Hard margin v.s. Soft margin - a lengthy series of experiments in which various parameters are tested This slide is courtesy of www.iro.umontreal.ca/~pift6080/documents/papers/svm_tutorial.ppt

Summary: Support Vector Machine 1. Large Margin Classifier Better generalization ability & less over-fitting 2. The Kernel Trick Map data points to higher dimensional space in order to make them linearly separable. Since only dot product is used, we do not need to represent the mapping explicitly.

Additional Resource http://www.kernel-machines.org/

LibSVM (best implementation for SVM) http://www.csie.ntu.edu.tw/~cjlin/libsvm/