Support Vector Machines part 2

Slides:

Advertisements

Similar presentations

Introduction to Support Vector Machines (SVM)

Advertisements

Support Vector Machines

Image classification Given the bag-of-features representations of images from different classes, how do we learn a model for distinguishing them?

INTRODUCTION TO Machine Learning 2nd Edition

Support Vector Machines and Kernels Adapted from slides by Tim Oates Cognition, Robotics, and Learning (CORAL) Lab University of Maryland Baltimore County.

1 Lecture 5 Support Vector Machines Large-margin linear classifier Non-separable case The Kernel trick.

Search Engines Information Retrieval in Practice All slides ©Addison Wesley, 2008.

Machine learning continued Image source:

SVMs Reprised. Administrivia I’m out of town Mar 1-3 May have guest lecturer May cancel class Will let you know more when I do...

Support Vector Machine

Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA.

Support Vector Machines (and Kernel Methods in general)

Support Vector Machines and Kernel Methods

Image classification Given the bag-of-features representations of images from different classes, how do we learn a model for distinguishing them?

Support Vector Machines (SVMs) Chapter 5 (Duda et al.)

Margins, support vectors, and linear programming Thanks to Terran Lane and S. Dreiseitl.

Sketched Derivation of error bound using VC-dimension (1) Bound our usual PAC expression by the probability that an algorithm has 0 error on the training.

Announcements  Project teams should be decided today! Otherwise, you will work alone.  If you have any question or uncertainty about the project, talk.

SVMs Finalized. Where we are Last time Support vector machines in grungy detail The SVM objective function and QP Today Last details on SVMs Putting it.

SVMs Reprised Reading: Bishop, Sec 4.1.1, 6.0, 6.1, 7.0, 7.1.

Support Vector Machines

Lecture 10: Support Vector Machines

Optimization Theory Primal Optimization Problem subject to: Primal Optimal Value:

SVMs, cont’d Intro to Bayesian learning. Quadratic programming Problems of the form Minimize: Subject to: are called “quadratic programming” problems.

This week: overview on pattern recognition (related to machine learning)

Copyright © 2001, Andrew W. Moore Support Vector Machines Andrew W. Moore Associate Professor School of Computer Science Carnegie Mellon University.

Support Vector Machines Mei-Chen Yeh 04/20/2010. The Classification Problem Label instances, usually represented by feature vectors, into one of the predefined.

1 CSC 4510, Spring © Paula Matuszek CSC 4510 Support Vector Machines 2 (SVMs)

Kernel Methods A B M Shawkat Ali 1 2 Data Mining ¤ DM or KDD (Knowledge Discovery in Databases) Extracting previously unknown, valid, and actionable.

Classifiers Given a feature representation for images, how do we learn a model for distinguishing features from different classes? Zebra Non-zebra Decision.

Kernels Usman Roshan CS 675 Machine Learning. Feature space representation Consider two classes shown below Data cannot be separated by a hyperplane.

CS 478 – Tools for Machine Learning and Data Mining SVM.

SVM – Support Vector Machines Presented By: Bella Specktor.

Support vector machine LING 572 Fei Xia Week 8: 2/23/2010 TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A 1.

Support Vector Machine Debapriyo Majumdar Data Mining – Fall 2014 Indian Statistical Institute Kolkata November 3, 2014.

Support Vector Machines. Notation Assume a binary classification problem. –Instances are represented by vector x   n. –Training examples: x = (x 1,

Final Exam Review CS479/679 Pattern Recognition Dr. George Bebis 1.

Text Classification using Support Vector Machine Debapriyo Majumdar Information Retrieval – Spring 2015 Indian Statistical Institute Kolkata.

Support Vector Machines Reading: Ben-Hur and Weston, “A User’s Guide to Support Vector Machines” (linked from class web page)

SVMs in a Nutshell.

Learning by Loss Minimization. Machine learning: Learn a Function from Examples Function: Examples: – Supervised: – Unsupervised: – Semisuprvised:

Introduction to Machine Learning Prof. Nir Ailon Lecture 5: Support Vector Machines (SVM)

1 Kernel Machines A relatively new learning methodology (1992) derived from statistical learning theory. Became famous when it gave accuracy comparable.

A Brief Introduction to Support Vector Machine (SVM) Most slides were from Prof. A. W. Moore, School of Computer Science, Carnegie Mellon University.

Support Vector Machines Reading: Textbook, Chapter 5 Ben-Hur and Weston, A User’s Guide to Support Vector Machines (linked from class web page)

SUPPORT VECTOR MACHINES

Neural networks and support vector machines

Support vector machines

PREDICT 422: Practical Machine Learning

Support Vector Machine

ECE 5424: Introduction to Machine Learning

ECE 5424: Introduction to Machine Learning

Support Vector Machines and Kernels

Geometrical intuition behind the dual problem

LINEAR AND NON-LINEAR CLASSIFICATION USING SVM and KERNELS

Pawan Lingras and Cory Butz

Support Vector Machines Introduction to Data Mining, 2nd Edition by

Support Vector Machines

Support Vector Machines

CS 2750: Machine Learning Support Vector Machines

CSSE463: Image Recognition Day 14

COSC 4335: Other Classification Techniques

Support vector machines

Support Vector Machines and Kernels

Class #212 – Thursday, November 12

Support vector machines

COSC 4368 Machine Learning Organization

Linear Discrimination

SVMs for Document Ranking

Support Vector Machines

Presentation transcript:

Support Vector Machines part 2 21 March 2013 Some slides from F. Bach and Z. Harchaoui

Motivation Max-margin classification Classification with kernels Left image: http://www.sussex.ac.uk/Users/christ/crs/ml/lec08a.html Right image: http://www.cs.helsinki.fi/group/smart/teaching/58308109/niissaloPrint.pdf

Primal and Dual formulations Maximize where Subject to these constraints:

Primal vs Dual Formulations n (# samples) params. Can efficiently handle very high dimensional data No need for explicit features (kernel trick) Is not very efficient for very large data sets Need to store the support vectors Primal d (data dimension) parameters Efficient when the number of samples is high (millions) – stochastic grad. desc. Easy on memory (store only w and b)

SVM Kernel Functions K(a,b)=(a . b +1)d is an example of an SVM Kernel Function Beyond polynomials there are other very high dimensional basis functions that can be made practical by finding the right Kernel Function Radial-Basis-style Kernel Function: Bandwidth: from linear classifier to NN methos Copyright © 2001, 2003, Andrew W. Moore

Kernel Tricks Replacing dot product with a kernel function Not all functions are kernel functions Need to be decomposable K(a,b) = (a)  (b) Could K(a,b) = (a-b)3 be a kernel function ? Could K(a,b) = (a-b)4 – (a+b)2 be a kernel function? Copyright © 2001, 2003, Andrew W. Moore

SVM for CSE 802 Project Copyright © 2001, 2003, Andrew W. Moore

SVM - FAQ Which formulation to use? How to set C? Which kernel to use? How to determine the kernel parameters? How to use SVM for multi-class problems (project)?

Multi-class / multi-label SVM Multi-class classification: each instance belong to one of the K classes Multi-label classification: each instance might belong to one or more members of the K classes

Multi-class / multi-label SVM For multi-class only!!!

OvsO vs OvsR OvR: K classifiers with n instances each OvO: K(K-1)/2 classifiers with O(2n/K) instances each (average) OvR: Data imbalance problem OvO: Hot to decide the winner

Data imbalance problem

A small experiment

A small experiment

SVM for CSE 802 Project

SVM for CSE 802 Project I have many feature or kernel parameter (type, bandwidth, degree) options and I am indecisive. I have many feature or kernel parameter (type, bandwidth, degree) options and I want to use them all somehow.

Gradient descend

Stochastic gradient descend

SVM - FAQ Which formulation to use? How to set C? Which kernel to use? How to determine the kernel parameters? How to use SVM for multi-class problems (project)?