Evaluating Classifiers Reading: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)An introduction to ROC analysis.

Slides:

Advertisements

Similar presentations

Classification.. continued. Prediction and Classification Last week we discussed the classification problem.. – Used the Naïve Bayes Method Today..we.

Advertisements

Evaluating Classifiers

Learning Algorithm Evaluation

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.

Lecture Notes for Chapter 4 (2) Introduction to Data Mining

Lecture 22: Evaluation April 24, 2010.

Lecture Notes for Chapter 4 Part III Introduction to Data Mining

The University of Texas at Austin, CS 395T, Spring 2008, Prof. William H. Press 1 Computational Statistics with Application to Bioinformatics Prof. William.

Assessing and Comparing Classification Algorithms Introduction Resampling and Cross Validation Measuring Error Interval Estimation and Hypothesis Testing.

Model Evaluation Metrics for Performance Evaluation

Cost-Sensitive Classifier Evaluation Robert Holte Computing Science Dept. University of Alberta Co-author Chris Drummond IIT, National Research Council,

Credibility: Evaluating what’s been learned. Evaluation: the key to success How predictive is the model we learned? Error on the training data is not.

Supervised classification performance (prediction) assessment Dr. Huiru Zheng Dr. Franscisco Azuaje School of Computing and Mathematics Faculty of Engineering.

© Vipin Kumar CSci 8980 Fall CSci 8980: Data Mining (Fall 2002) Vipin Kumar Army High Performance Computing Research Center Department of Computer.

On Comparing Classifiers: Pitfalls to Avoid and Recommended Approach Published by Steven L. Salzberg Presented by Prakash Tilwani MACS 598 April 25 th.

Notes on Measuring the Performance of a Binary Classifier David Madigan.

Introduction to Machine Learning Approach Lecture 5.

Today Evaluation Measures Accuracy Significance Testing

Classification II (continued) Model Evaluation

CSCI 347 / CS 4206: Data Mining Module 06: Evaluation Topic 01: Training, Testing, and Tuning Datasets.

Evaluating Classifiers

SVMLight SVMLight is an implementation of Support Vector Machine (SVM) in C. Download source from :

Link Reconstruction from Partial Information Gong Xiaofeng, Li Kun & C. H. Lai

1 Statistics 202: Statistical Aspects of Data Mining Professor David Mease Tuesday, Thursday 9:00-10:15 AM Terman 156 Lecture 11 = Finish ch. 4 and start.

1 Evaluating Model Performance Lantz Ch 10 Wk 5, Part 2 Right – Graphing is often used to evaluate results from different variations of an algorithm. Depending.

Error estimation Data Mining II Year Lluís Belanche Alfredo Vellido.

Evaluating Hypotheses Reading: Coursepack: Learning From Examples, Section 4 (pp )

Data Analysis 1 Mark Stamp. Topics  Experimental design o Training set, test set, n-fold cross validation, thresholding, imbalance, etc.  Accuracy o.

Experimental Evaluation of Learning Algorithms Part 1.

Classification Performance Evaluation. How do you know that you have a good classifier? Is a feature contributing to overall performance? Is classifier.

Machine learning system design Prioritizing what to work on

F. Provost and T. Fawcett. Confusion Matrix 2Bitirgen - CS678.

CSSE463: Image Recognition Day 11 Lab 4 (shape) tomorrow: feel free to start in advance Lab 4 (shape) tomorrow: feel free to start in advance Test Monday.

Evaluating Results of Learning Blaž Zupan

Computational Intelligence: Methods and Applications Lecture 16 Model evaluation and ROC Włodzisław Duch Dept. of Informatics, UMK Google: W Duch.

ASSESSING LEARNING ALGORITHMS Yılmaz KILIÇASLAN. Assessing the performance of the learning algorithm A learning algorithm is good if it produces hypotheses.

Model Evaluation l Metrics for Performance Evaluation –How to evaluate the performance of a model? l Methods for Performance Evaluation –How to obtain.

ASSESSING LEARNING ALGORITHMS Yılmaz KILIÇASLAN. Assessing the performance of the learning algorithm A learning algorithm is good if it produces hypotheses.

Practical Issues of Classification Underfitting and Overfitting –Training errors –Generalization (test) errors Missing Values Costs of Classification.

Machine Learning Tutorial-2. Recall, Precision, F-measure, Accuracy Ch. 5.

Classification Evaluation. Estimating Future Accuracy Given available data, how can we reliably predict accuracy on future, unseen data? Three basic approaches.

GENDER AND AGE RECOGNITION FOR VIDEO ANALYTICS SOLUTION PRESENTED BY: SUBHASH REDDY JOLAPURAM.

1 Performance Measures for Machine Learning. 2 Performance Measures Accuracy Weighted (Cost-Sensitive) Accuracy Lift Precision/Recall –F –Break Even Point.

CSSE463: Image Recognition Day 11 Due: Due: Written assignment 1 tomorrow, 4:00 pm Written assignment 1 tomorrow, 4:00 pm Start thinking about term project.

Evaluating Classification Performance

Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.

A Brief Introduction and Issues on the Classification Problem Jin Mao Postdoc, School of Information, University of Arizona Sept 18, 2015.

Notes on HW 1 grading I gave full credit as long as you gave a description, confusion matrix, and working code Many people’s descriptions were quite short.

Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson.

ROC curve estimation. Index Introduction to ROC ROC curve Area under ROC curve Visualization using ROC curve.

Chapter 5: Credibility. Introduction Performance on the training set is not a good indicator of performance on an independent set. We need to predict.

Evaluation of Learning Models Evgueni Smirnov. Overview Motivation Metrics for Classifier’s Evaluation Methods for Classifier’s Evaluation Comparing Data.

Evaluating Classifiers. Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)

Supervise Learning. 2 What is learning? “Learning denotes changes in a system that... enable a system to do the same task more efficiently the next time.”

7. Performance Measurement

Evaluating Classifiers

Performance Evaluation 02/15/17

CSSE463: Image Recognition Day 11

Classification with Perceptrons Reading:

Data Mining Classification: Alternative Techniques

CSSE463: Image Recognition Day 11

Evaluating Classifiers (& other algorithms)

Learning Algorithm Evaluation

Model Evaluation and Selection

CSSE463: Image Recognition Day 11

CSSE463: Image Recognition Day 11

Evaluating Classifiers

ECE – Pattern Recognition Lecture 8 – Performance Evaluation

Presentation transcript:

Evaluating Classifiers Reading: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)An introduction to ROC analysis

Evaluating Classifiers What we want: classifier that best predicts unseen data Assumption: Data is “iid” (independently and identically distributed)

Accuracy and Error Rate Accuracy = fraction of correct classifications on unseen data (test set) Error rate = 1 − Accuracy

How to use available data to best measure accuracy? Split data into training and test sets. But how to split? Too little training data: Don’t get optimal classifier Too little test data: Measured accuracy is not correct

One solution: “k-fold cross validation” Each example is used both as a training instance and as a test instance. Split data into k disjoint parts: S 1, S 2,..., S k. For i = 1 to k Select S i to be the test set. Train on the remaining data, test on S i, to obtain accuracy A i. Report as the final accuracy.

“Confusion matrix” for a given class c Actual Predicted (or “classified”) PositiveNegative (in class c)(not in class c) Positive (in class c) TruePositiveFalseNegative Negative (not in class c)FalsePositiveTrueNegative Evaluating classification algorithms

Example: “A” vs. “B” Results from Perceptron: Instance Class Perception Output 1 “A”−1 2 “A”+1 3 “A”+1 4 “A”−1 5 “B”+1 6 “B”−1 7 “B”−1 1 “B”−1 Accuracy: ActualPredicted PositiveNegative Positive Negative Assume “A” is positive class Confusion Matrix

Evaluating classification algorithms “Confusion matrix” for a given class c Actual Predicted (or “classified”) PositiveNegative (in class c)(not in class c) Positive (in class c) TruePositiveFalseNegative Negative (not in class c)FalsePositiveTrueNegative Type 2 error Type 1 error

Precision and Recall All instances in test set Positive Instances Negative Instances

Precision and Recall All instances in test set Instances classified as positive Positive Instances Negative Instances

Instances classified as negative Precision and Recall All instances in test set Instances classified as positive Positive Instances Negative Instances Recall = Sensitivity = True Positive Rate

Example: “A” vs. “B” Results from Perceptron: Instance Class Perception Output 1 “A”−1 2 “A”+1 3 “A”+1 4 “A”−1 5 “B”+1 6 “B”−1 7 “B”−1 1 “B”−1 Assume “A” is positive class

Results of classifier ThresholdAccuracyPrecisionRecall ∞ Creating a Precision vs. Recall Curve

14 (“sensitivity”) (1  “specificity”)

15

Results of classifier ThresholdAccuracyTPRFPR ∞ Creating a ROC Curve

Precision/Recall versus ROC curves 17

18

19

20 There are various algorithms for computing AOC (often built into statistical software) – we won’t cover this in this class.

How to create a ROC curve for a perceptron Run your perceptron on each instance in the test data, without doing the sgn step: Get the range [min, max] of your scores Divide the range into about 200 thresholds, including −∞ and max For each threshold, calculate TPR and FPR Plot TPR (y-axis) vs. FPR (x-axis)