1. Last lecture summary (SVM)

2. Support Vector Machine Supervised algorithm Works both as – classifier (binary) – regressor De facto better linear classification Two main ingrediences: – maximum margin – kernel functions

3. Maximum margin Which line is best?

4. Define the margin of a linear classifier as the width that the boundary could be increased by before hitting a datapoint. The maximum margin linear classifier is the optimum linear classifier. This is the simplest kind of SVM (linear SVM) Maximum margin intuitively feels safest. Only support vectors are important. Works very well.

5. The decision boundary is found by constrained quadratic optimization. The solution is found in the form Only points on the margin (i.e. support vectors x i ) have α i > 0. Lagrange multiplier

6. w does not to be explicitly formed, because: Training SVM: find the sets of the parameters α i and b. Classification with SVM:

7. soft margin – Allows misclassification errors. – i.e. misclassified points are allowed to be inside the margin. – The penalty to classification errors is given by the capacity parameter C (user adjustable parameter). – Large C – a high penalty to classification errors. – Decrease in C: points move inside margin.

8. CSE 802. Prepared by Martin Law

9. Kernel functions Soft margin introduces the possibility to linearly classify the linearly non-separable data sets. What else could be done? Can we propose an approach generating non-linear classification boundary just by extending the linear classifier machinery?

11. X

15. Kernels Linear (dot) kernel Polynomial – simple, efficient for non-linear relationships – d – degree Gaussian

16. Finishing SVM

17. SVM parameters Training sets the parameters α i and b. The SVM has another set of parameters called hyperparameters. – The soft margin constant C. – Any parameters the kernel function depends on linear kernel – no hyperparameter (except for C) polynomial – degree Gaussian – width of Gaussian

18. So which kernel and which parameters should I use? The answer is data-dependent. Several kernels should be tried. Try linear kernel first and see, if the classification can be improved with nonlinear kernels (tradeoff between quality of the kernel and the number of dimensions). Select kernel + parameters + C by crossvalidation.

19. Computational aspects Classification of new samples is very quick, training is longer (reasonably fast for thousands of samples). Linear kernel – scales linearly. Nonlinear kernels – scale quadratically.

20. Multiclass SVM SVM is defined for binary classification. How to predict more than two classes (multiclass)? Simplest approach: decompose the multiclass problem into several binary problems and train several binary SVM’s.

21. 1/2 1 1/31/42/32/32/43/4 1 1 1 3 4 4

22. 1/rest2/rest3/rest4/rest

23. Resources SVM and Kernels for Comput. Biol., Ratsch et al., PLOS Comput. Biol., 4 (10), 1-10, 2008 What is a support vector machine, W. S. Noble, Nature Biotechnology, 24 (12), 1565-1567, 2006 A tutorial on SVM for pattern recognition, C. J. C. Burges, Data Mining and Knowledge Discovery, 2, 121-167, 1998 A User’s Guide to Support Vector Machines, Asa Ben-Hur, Jason Weston

24. http://support-vector-machines.org/ http://www.kernel-machines.org/ http://www.support-vector.net/ – companion to the book An Introduction to Support Vector Machines by Cristianini and Shawe-Taylor http://www.kernel-methods.net/ – companion to the book Kernel Methods for Pattern Analysis by Shawe-Taylor and Cristianini http://www.learning-with-kernels.org/ – Several chapters on SVM from the book Learning with Kernels by Scholkopf and Smola are available from this site

25. Software SVM light – one of the most widely used SVM package. fast optimization, can handle very large datasets, very efficient implementation of the leave–one–out cross-validation, C++ code SVM struct - can model complex data, such as trees, sequences, or sets LIBSVM – multiclass, weighted SVM for unbalanced data, cross-validation, automatic model selection, C++, Java

26. Naïve Bayes Classifier

27. Example – Play Tennis

28. Example – Learning Phase OutlookPlay=YesPlay=No Sunny 2/93/5 Overcast 4/90/5 Rain 3/92/5 TemperaturePlay=YesPlay=No Hot 2/92/5 Mild 4/92/5 Cool 3/91/5 HumidityPlay=YesPlay=No High 3/94/5 Normal 6/91/5 WindPlay=YesPlay=No Strong 3/93/5 Weak 6/92/5 P(Play=Yes) = 9/14P(Play=No) = 5/14 P(Outlook=Sunny|Play=Yes) = 2/9

29. Example - prediction Answer this question: “Will we play tennis given that it’s cool but sunny, humidity is high and it is blowing a strong wind?” i.e., predict this new instace: x’=(Outl=Sunny, Temp=Cool, Hum=High, Wind=Strong) Good strategy is to predict arg max P(Y|cool,sunny,high,strong) where Y is Yes or No.

30. Example - Prediction x’=(Outl=Sunny, Temp=Cool, Hum=High, Wind=Strong) Look up tables P(Outl=Sunny|Play=No) = 3/5 P(Temp=Cool|Play=No) = 1/5 P(Hum=High|Play=No) = 4/5 P(Wind=Strong|Play=No) = 3/5 P(Play=No) = 5/14 P(Outl=Sunny|Play=Yes) = 2/9 P(Temp=Cool|Play=Yes) = 3/9 P(Hum=High|Play=Yes) = 3/9 P(Wind=Strong|Play=Yes) = 3/9 P(Play=Yes) = 9/14 P(Yes|x’): [P(Sunny|Yes)P(Cool|Yes)P(High|Yes)P(Strong|Yes)]P(Play=Yes) = 0.0053 P(No|x’): [P(Sunny|No) P(Cool|No)P(High|No)P(Strong|No)]P(Play=No) = 0.0206 Given the fact P(Yes| x ’) < P(No| x ’), we label x ’ to be “No”.

31. Another Application Digit Recognition X 1,…,X n  {0,1} (Black vs. White pixels) Y  {5,6} (predict whether a digit is a 5 or a 6) Classifier 5

32. Bayes Rule So how do we compute posterior probability that the image represents a 5 given its pixels? Why did this help? Well, we think that we might be able to specify how features are “generated” by the class label (i.e. we will try to compute likelihood). Normalization Constant LikelihoodPrior Posterior

33. Let’s expand this for our digit recognition task: To classify, we’ll simply compute these two probabilities and predict based on which one is greater. For the Bayes classifier, we need to “learn” two functions, the likelihood and the prior.

34. Learning prior Let us assume training examples are generated by drawing instances at random from an unknown underlying distribution P(Y), then allow a teacher to label this example with its Y value. A hundred independently drawn training examples will usually suffice to obtain a reasonable estimate of P(Y).

35. Learning likelihood

36. So this corresponds to two distinct parameters for each of the distinct instances in the instance space for X. Worse yet, to obtain reliable estimates of each of these parameters, we will need to observe each of these distinct instances multiple times. For example, if X is a vector containing 30 boolean features, then we will need to estimate more than 3 billion parameters!

38. The problem with explicitly modeling P(X 1,…,X n |Y) is that there are usually way too many parameters: – We’ll run out of space. – We’ll run out of time. – And we’ll need tons of training data (which is usually not available).

39. The Naïve Bayes Model

40. Naïve Bayes Training MNIST Training Data

41. Naïve Bayes Training Training in Naïve Bayes is easy: – Estimate P(Y=v) as the fraction of records with Y=v – Estimate P(X i =u|Y=v) as the fraction of records with Y=v for which X i =u

42. In practice, some of these counts can be zero Fix this by adding “virtual” counts: This is called Smoothing. Naïve Bayes Training

43. For binary digits, training amounts to averaging all of the training fives together and all of the training sixes together.

44. Naïve Bayes Classification

45. Assorted remarks What’s nice about Naïve Bayes is that it returns probabilities – These probabilities can tell us how confident the algorithm is – So… don’t throw away these probabilities! Naïve Bayes assumption is almost never true – Still… Naïve Bayes often performs surprisingly well even when its assumptions do not hold. – Very good method in text processing.

46. Binary classifier performance

47. Confusion matrix TPTrue Positives – is positive and is classified as positive TNTrue Negatives – is negative and is classified as negative FPFalse Positives – is negative, but is classified as positive FNFalse Negatives – is positive, but is classified as negative KnownLabel positivenegative Predicted positive TPFP Label negative FNTN also called a contingency table

48. Accuracy Accuracy = (TP + TN) / (TP + TN + FP + FN) KnownLabel positivenegative Predicted positive TPFP Label negative FNTN

49. Information retrieval (IR) A query by the user – to find the documents in the database. IR systems allow to narrow down the set of documents that are relevant to a particular problem.

50. documents containing what I am looking for documents not containing what I am looking for

51. TPFP FN Precision = TP / (TP + FP) Recall = TP / (TP + FN) How many of the things I consider to be true are actually true? How much of the true things do I find? TN

52. Precision A measure of exactness A perfect precision score of 1.0 means that every result retrieved by a search was relevant. But says nothing about whether all relevant documents were retrieved. TPFP

53. Recall A measure of completeness A perfect recall score of 1.0 means that all relevant documents were retrieved by the search. But says nothing about how many irrelevant documents were also retrieved. TP FN

54. Precission-Recall tradeoff Returning all documents lead to the perfect recall of 1.0. – i.e. all relevant documents are present in the returned set However, precission is not that great, as not every result is relevant. Apparently, the relationship between them is inverse – it is possible to increase one at the cost of reducing the other. They are not discussed in isolation. – Either values for one measure are compared for a fixed level at the other measure (e.g. precision at the recall level of 0.75) – Combine both into the F-measure.

55. F-measure Common F 1 measure General F β measure β - relative value of precision –β = 1 – weight precision and recall by the same amount –β < 1 – more weight on precision –β > 1 – more weight on recall

56. Sensitivity & Specificity Measure how ‘good’ a test is at detecting binary features of interest (disease/no disease). There are 100 patients, 30 have disease A. A test designed to identify who has the disease and who does not. We want to evaluate how good the test is.

57. Sensitivity & Specificity Disease + Disease - Test + 252 Test - 568

58. Sensitivity & Specificity 25/30 sensitivity 68/70 specificity Disease + Disease - Total Test + 25227 Test - 56873 Total 3070100

59. Sensitivity measures the proportion of actual positives which are correctly identified as such (e.g. the percentage of sick people who are identified as sick). TP/(TP + FN) Specificity measures the proportion of negatives which are correctly identified (e.g. the percentage of healthy people who are identified as healthy). TN/(TN + FP)

60. Performance Evaluation Precision, Positive Predictive Value (PPV) TP / (TP + FP) Recall, Sensitivity, True Positive Rate (TPR), Hit rate TP / P = TP/(TP + FN) False Positive Rate (FPR), Fall-out FP / N = FP / (FP + TN) Specificity, True Negative Rate (TNR) TN / (TN + FP) = 1 - FPR Accuracy (TP + TN) / (TP + TN + FP + FN)

61. Types of classifiers A discrete (crisp) classifier – Output is only a class label, e.g. decision tree. A soft classifier – Yield a probability (score, confidence) for the given pattern. – Number representing the degree to which an instance is a member of a class. – Use threshold to assign to (+) or to (-) class. – e. g. SVM, NN, naïve Bayes.

62. ROC Graph Receiver Operating Characteristics. Plot TPR vs. FPR Sensitivity vs. (1 – Specificity). TPR is on the Y axis, FPR on the X axis. An ROC graph depicts relative trade-offs between benefits (true positives) and costs (false positives).

63. random guess worse better perfect classification Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27, 861–874. never issue positive classification always issue positive classification 0.5, 0.5 perfect classification

64. A is more conservative than B. conservative classifiers liberal classifiers They make positive classifications only with strong evidence so they make few false positive errors, but they often have low true positive rates as well They make positive classifications with weak evidence so they classify nearly all positives correctly, but they often have high false positive rates

65. ROC Curve Fawcet, ROC Graphs: Notes and Practical Considerations for Researchers

66. lowering threshold corresponds to moving from the conservative to the liberal areas of the graph Fawcet, ROC Graphs: Notes and Practical Considerations for Researchers

68. We have 6 positives and 4 negatives in a test set. All are scored equally. Fawcet, ROC Graphs: Notes and Practical Considerations for Researchers

69. ROC curves are insensitive to the changes in class distribution Fawcet, ROC Graphs: Notes and Practical Considerations for Researchers

71. AUC To compare classifiers we may want to reduce ROC performance to a single scalar value representing expected performance. A common method is to calculate the area under the ROC curve, abbreviated AUC. Its value will always be between 0 and 1.0. Random guessing has an area 0.5. Any realistic classifier should have an AUC between 0.5 and 1.0.

72. The AUC has an important statistical property: the AUC of a classifier is equivalent to the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance. Classier B is generally better than A, it has higher AUC. However, with the exception at FPR > 0.55 where A has slight advantage. So it is possible for a high-AUC classifier to perform worse in a specific region of ROC space than a low-AUC classifier. But in practice the AUC performs very well and is often used when a general measure of predictiveness is desired.