Computational Learning Theory

Slides:

Advertisements

Similar presentations

Statistical Machine Learning- The Basic Approach and Current Research Challenges Shai Ben-David CS497 February, 2007.

Advertisements

Introduction to Support Vector Machines (SVM)

Lecture 9 Support Vector Machines

SVM - Support Vector Machines A new classification method for both linear and nonlinear data It uses a nonlinear mapping to transform the original training.

Classification / Regression Support Vector Machines

Support Vector Machines Elena Mokshyna Odessa 2014.

1 Lecture 5 Support Vector Machines Large-margin linear classifier Non-separable case The Kernel trick.

SVM—Support Vector Machines

Support Vector Machines (and Kernel Methods in general)

Support Vector Machines and Kernel Methods

Support Vector Machines (SVMs) Chapter 5 (Duda et al.)

The Nature of Statistical Learning Theory by V. Vapnik

Computational Learning Theory PAC IID VC Dimension SVM Kunstmatige Intelligentie / RuG KI2 - 5 Marius Bulacu & prof. dr. Lambert Schomaker.

Northwestern University Winter 2007 Machine Learning EECS Machine Learning Lecture 13: Computational Learning Theory.

Support Vector Classification (Linearly Separable Case, Primal) The hyperplanethat solves the minimization problem: realizes the maximal margin hyperplane.

Classification Problem 2-Category Linearly Separable Case A- A+ Malignant Benign.

Sketched Derivation of error bound using VC-dimension (1) Bound our usual PAC expression by the probability that an algorithm has 0 error on the training.

A Kernel-based Support Vector Machine by Peter Axelberg and Johan Löfhede.

1 Computational Learning Theory and Kernel Methods Tianyi Jiang March 8, 2004.

SVMs Finalized. Where we are Last time Support vector machines in grungy detail The SVM objective function and QP Today Last details on SVMs Putting it.

Lecture outline Support vector machines. Support Vector Machines Find a linear hyperplane (decision boundary) that will separate the data.

SVM Support Vectors Machines

Support Vector Machines

What is Learning All about ?  Get knowledge of by study, experience, or being taught  Become aware by information or from observation  Commit to memory.

Lecture 10: Support Vector Machines

Statistical Learning Theory: Classification Using Support Vector Machines John DiMona Some slides based on Prof Andrew Moore at CMU:

Support Vector Machines Piyush Kumar. Perceptrons revisited Class 1 : (+1) Class 2 : (-1) Is this unique?

Linear hyperplanes as classifiers Usman Roshan. Hyperplane separators.

Based on: The Nature of Statistical Learning Theory by V. Vapnick 2009 Presentation by John DiMona and some slides based on lectures given by Professor.

Machine Learning1 Machine Learning: Summary Greg Grudic CSCI-4830.

CS 8751 ML & KDDSupport Vector Machines1 Support Vector Machines (SVMs) Learning mechanism based on linear programming Chooses a separating plane based.

Support Vector Machine (SVM) Based on Nello Cristianini presentation

Support Vector Machines Reading: Ben-Hur and Weston, “A User’s Guide to Support Vector Machines” (linked from class web page)

1 Machine Learning: Lecture 8 Computational Learning Theory (Based on Chapter 7 of Mitchell T.., Machine Learning, 1997)

Kernels Usman Roshan CS 675 Machine Learning. Feature space representation Consider two classes shown below Data cannot be separated by a hyperplane.

Computational Learning Theory IntroductionIntroduction The PAC Learning FrameworkThe PAC Learning Framework Finite Hypothesis SpacesFinite Hypothesis Spaces.

Kernel Methods: Support Vector Machines Maximum Margin Classifiers and Support Vector Machines.

CSE4334/5334 DATA MINING CSE4334/5334 Data Mining, Fall 2014 Department of Computer Science and Engineering, University of Texas at Arlington Chengkai.

Support Vector Machine Debapriyo Majumdar Data Mining – Fall 2014 Indian Statistical Institute Kolkata November 3, 2014.

Support Vector Machines Tao Department of computer science University of Illinois.

Final Exam Review CS479/679 Pattern Recognition Dr. George Bebis 1.

Goal of Learning Algorithms  The early learning algorithms were designed to find such an accurate fit to the data.  A classifier is said to be consistent.

Text Classification using Support Vector Machine Debapriyo Majumdar Information Retrieval – Spring 2015 Indian Statistical Institute Kolkata.

Kernel Methods: Support Vector Machines Maximum Margin Classifiers and Support Vector Machines.

Machine Learning Chapter 7. Computational Learning Theory Tom M. Mitchell.

SVMs in a Nutshell.

Generalization Error of pac Model  Let be a set of training examples chosen i.i.d. according to  Treat the generalization error as a r.v. depending on.

FUZZ-IEEE Kernel Machines and Additive Fuzzy Systems: Classification and Function Approximation Yixin Chen and James Z. Wang The Pennsylvania State.

Support Vector Machine (SVM) Presented by Robert Chen.

Introduction to Machine Learning Prof. Nir Ailon Lecture 5: Support Vector Machines (SVM)

1 Kernel Machines A relatively new learning methodology (1992) derived from statistical learning theory. Became famous when it gave accuracy comparable.

Support Vector Machines Reading: Textbook, Chapter 5 Ben-Hur and Weston, A User’s Guide to Support Vector Machines (linked from class web page)

Support Vector Machines (SVMs) Chapter 5 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis.

1 CS 391L: Machine Learning: Computational Learning Theory Raymond J. Mooney University of Texas at Austin.

STATISTICAL LEARNING THEORY & CLASSIFICATIONS BASED ON SUPPORT VECTOR MACHINES presenter: Xipei Liu Vapnik, Vladimir. The nature of statistical.

PREDICT 422: Practical Machine Learning

LINEAR AND NON-LINEAR CLASSIFICATION USING SVM and KERNELS

Support Vector Machines Introduction to Data Mining, 2nd Edition by

Support Vector Machines

Pattern Recognition CS479/679 Pattern Recognition Dr. George Bebis

Computational Learning Theory

CSSE463: Image Recognition Day 14

CSSE463: Image Recognition Day 14

LEARNING Chapter 18b and Chuck Dyer

Computational Learning Theory

CSSE463: Image Recognition Day 15

CSSE463: Image Recognition Day 15

CSCE833 Machine Learning Lecture 9 Linear Discriminant Analysis

CSSE463: Image Recognition Day 15

Linear Discrimination

Presentation transcript:

Computational Learning Theory PAC IID VC Dimension SVM Marius Bulacu Kunstmatige Intelligentie / RuG

computational learning theory The Problem Why does learning work? How do we know that the learned hypothesis h is close to the target function f if we do not know what f is? answer provided by computational learning theory

Probably Approximately Correct The Answer Any hypothesis h that is consistent with a sufficiently large number of training examples is unlikely to be seriously wrong. Therefore it must be: Probably Approximately Correct PAC

The Stationarity Assumption The training and test sets are drawn randomly from the same population of examples using the same probability distribution. Therefore training and test data are Independently and Identically Distributed IID “the future is like the past”

How many examples are needed? Probability of existence of a wrong hypothesis consistent with all examples Size of hypothesis space Number of examples Probability that h and f disagree on an example Sample complexity

Formal Derivation e f H (the set of all possible hypothese) HBAD (the set of “wrong” hypotheses) e f

What if hypothesis space is infinite? Can’t use our result for finite H Need some other measure of complexity for H Vapnik-Chervonenkis dimension

SVM (1): Kernels Kernels Polynomial Radial basis Sigmoid Implicit mapping to a higher dimensional space where linear separation is possible. f3 f2 f2 f1 f1 Complicated separation boundary Simple separation boundary: Hyperplane

SVM (2): Max Margin Good generalization Support vectors f2 “Best” Separating Hyperplane f1 Max Margin From all the possible separating hyperplanes, select the one that gives Max Margin. Solution found by Quadratic Optimization – “Learning”.