Fun with Hyperplanes: Perceptrons, SVMs, and Friends Adapted from slides by Ryan Gabbard CIS 391 – Introduction to Artificial Intelligence

Universal Machine Learning Diagram Naïve Bayes Classifiers are one example CIS 391 - Intro to AI

Generative v. Discriminative Models Generative question: “How can we model the joint distribution of the classes and the features?” Why waste energy on stuff we don’t care about? Let’s optimize the job we’re trying to do directly! Discriminative question: “What features distinguish the classes from one another?” CIS 391 - Intro to AI

chart from MIT tech report #507, Tony Jebara Example Modeling what sort of bizarre distribution produced these training points is hard, but distinguishing the classes is a piece of cake! chart from MIT tech report #507, Tony Jebara CIS 391 - Intro to AI

Linear Classification CIS 391 - Intro to AI

Why bother with this weird representation? Representing Lines How do we represent a line? In general a hyperplane is defined by Why bother with this weird representation? CIS 391 - Intro to AI

Projections alternate intuition: recall the dot product of two vectors is simply the product of their lengths and the cosine of the angle between them CIS 391 - Intro to AI

Now classification is easy! But... how do we learn this mysterious model vector? CIS 391 - Intro to AI

Perceptron Learning Algorithm CIS 391 - Intro to AI

Perceptron Update Example I CIS 391 - Intro to AI

Perceptron Update Example II CIS 391 - Intro to AI

Properties of the Simple Perceptron You can prove that If it’s possible to separate the data with a hyperplane (i.e. if it’s linearly separable), Then the algorithm will converge to that hyperplane. But what if it isn’t? Then perceptron is very unstable and bounces all over the place CIS 391 - Intro to AI

Voted Perceptron Works just like a regular perceptron, except you keep track of all the intermediate models you created When you want to classify something, you let each of the (many, many) models vote on the answer and take the majority CIS 391 - Intro to AI

Properties of Voted Perceptron Simple! Much better generalization performance than regular perceptron For later: (almost as good as SVMs) For later: Can use the ‘kernel trick’ Training as fast as regular perceptron But run-time is slower CIS 391 - Intro to AI

Averaged Perceptron Extremely simple! Return as your final model the average of all your intermediate models Approximation to voted perceptron Nearly as fast to train and exactly as fast to run as regular perceptron CIS 391 - Intro to AI

What’s wrong with these hyperplanes? CIS 391 - Intro to AI

They’re unjustifiably biased! CIS 391 - Intro to AI

A less biased choice CIS 391 - Intro to AI

Margin The margin is the distance to closest point in the training data We tend to get better generalization to unseen data if we choose the separating hyperplane which maximizes the margin CIS 391 - Intro to AI

Support Vector Machines Another learning method which explicitly calculates the maximum margin hyperplane by solving a gigantic quadratic programming problem. Generally considered the highest-performing current machine learning technique. But it’s relatively slow and very complicated. CIS 391 - Intro to AI

Margin-Infused Relaxed Algorithm (MIRA) Multiclass; each class has a prototype vector Classify an instance by choosing the class whose prototype vector has the greatest dot product with the instance During training, when updating make the ‘smallest’ (in a sense) change to the prototype vectors which guarantees correct classification by a minimum margin Pays attention to the margin directly CIS 391 - Intro to AI

What if it isn’t separable? CIS 391 - Intro to AI

Project it to someplace where it is! CIS 391 - Intro to AI

Kernel Trick If our data isn’t linearly separable, we can define a projection to map it into a much higher dimensional feature space where it is. For some algorithms where everything can be expressed as the dot products of instances (SVM, voted perceptron, MIRA) this can be done efficiently using something called the `kernel trick’ CIS 391 - Intro to AI