ROC Curves and Operating Points

Slides:

Advertisements

Similar presentations

Classification.. continued. Prediction and Classification Last week we discussed the classification problem.. – Used the Naïve Bayes Method Today..we.

Advertisements

Imbalanced data David Kauchak CS 451 – Fall 2013.

Learning Algorithm Evaluation

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.

Curva ROC figuras esquemáticas Curva ROC figuras esquemáticas Prof. Ivan Balducci FOSJC / Unesp.

Lecture 22: Evaluation April 24, 2010.

Assessing and Comparing Classification Algorithms Introduction Resampling and Cross Validation Measuring Error Interval Estimation and Hypothesis Testing.

Classification and risk prediction

Model Evaluation Metrics for Performance Evaluation

CAP and ROC curves.

Performance measures Morten Nielsen, CBS, BioCentrum, DTU.

Decision Theory Naïve Bayes ROC Curves

Performance Evaluation in Computer Vision Kyungnam Kim Computer Vision Lab, University of Maryland, College Park.

ROC Curve and Classification Matrix for Binary Choice Professor Thomas B. Fomby Department of Economics SMU Dallas, TX February, 2015.

Notes on Measuring the Performance of a Binary Classifier David Madigan.

Today Evaluation Measures Accuracy Significance Testing

CSCI 347 / CS 4206: Data Mining Module 06: Evaluation Topic 01: Training, Testing, and Tuning Datasets.

Evaluating Classifiers

1 Evaluating Model Performance Lantz Ch 10 Wk 5, Part 2 Right – Graphing is often used to evaluate results from different variations of an algorithm. Depending.

Data Analysis 1 Mark Stamp. Topics  Experimental design o Training set, test set, n-fold cross validation, thresholding, imbalance, etc.  Accuracy o.

Prediction of Malignancy of Ovarian Tumors Using Least Squares Support Vector Machines C. Lu 1, T. Van Gestel 1, J. A. K. Suykens 1, S. Van Huffel 1, I.

Classification Performance Evaluation. How do you know that you have a good classifier? Is a feature contributing to overall performance? Is classifier.

EMBC2001 Using Artificial Neural Networks to Predict Malignancy of Ovarian Tumors C. Lu 1, J. De Brabanter 1, S. Van Huffel 1, I. Vergote 2, D. Timmerman.

Model Evaluation. CRISP-DM CRISP-DM Phases Business Understanding – Initial phase – Focuses on: Understanding the project objectives and requirements.

Introduction Use machine learning and various classifying techniques to be able to create an algorithm that can decipher between spam and ham s. .

Evaluating Results of Learning Blaž Zupan

Model Evaluation l Metrics for Performance Evaluation –How to evaluate the performance of a model? l Methods for Performance Evaluation –How to obtain.

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 5: Credibility: Evaluating What’s Been Learned.

Classification Evaluation. Estimating Future Accuracy Given available data, how can we reliably predict accuracy on future, unseen data? Three basic approaches.

ROC curve estimation. Index Introduction to ROC ROC curve Area under ROC curve Visualization using ROC curve.

Spam Filtering Using Statistical Data Compression Models Andrej Bratko, Bogdan Filipič, Gordon V. Cormack, Thomas R. Lynam, Blaž Zupan Journal of Machine.

Chapter 5: Credibility. Introduction Performance on the training set is not a good indicator of performance on an independent set. We need to predict.

Data Analytics CMIS Short Course part II Day 1 Part 4: ROC Curves Sam Buttrey December 2015.

Performance measures Morten Nielsen, CBS, Department of Systems Biology, DTU.

Evaluation of Learning Models Evgueni Smirnov. Overview Motivation Metrics for Classifier’s Evaluation Methods for Classifier’s Evaluation Comparing Data.

Evaluating Classifiers. Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)

7. Performance Measurement

Evolving Decision Rules (EDR)

Evaluating Classifiers

Evaluation – next steps

Performance Evaluation 02/15/17

Modelling in Physical Geography Martin Mergili, University of Vienna

CSSE463: Image Recognition Day 11

CSE 4705 Artificial Intelligence

Evaluating Results of Learning

Dipartimento di Ingegneria «Enzo Ferrari»,

Support Vector Machines (SVM)

Data Mining Classification: Alternative Techniques

Features & Decision regions

Advanced Analytics. Advanced Analytics What is Machine Learning?

CSSE463: Image Recognition Day 11

Evaluating Classifiers (& other algorithms)

Image Classification via Attribute Detection

Approaching an ML Problem

Model Evaluation and Selection

Evaluating Models Part 1

Overfitting and Underfitting

Basics of ML Rohan Suri.

Evaluating Classifiers

CSSE463: Image Recognition Day 11

Roc curves By Vittoria Cozza, matr

CSSE463: Image Recognition Day 11

Evaluating Classifiers

ECE – Pattern Recognition Lecture 8 – Performance Evaluation

Logistic Regression Geoff Hulten.

ROC Curves and Operating Points

Information Organization: Evaluation of Classification Performance

Presentation transcript:

ROC Curves and Operating Points Geoff Hulten

Classifications and Probability Estimates Logistic regression produces a score (probability estimate) Use threshold to produce classification What happens if you vary the threshold?

Example of Changing Thresholds False Positive Rate 33% False Negative Rate 0% Score Prediction Y .25 .45 .55 1 .67 .82 .95 Threshold = .6 False Positive Rate 33% False Negative Rate Threshold = .7 False Positive Rate 0% False Negative Rate 33%

ROC Curve

Comparing Models with ROC Curves

More ROC Comparisons

Some Common ROC Problems

Precision Recall Curves – PR Curves Incremental Classifications More Accurate Everything Classified Correctly Incremental Classifications Less Accurate First Set of Mistakes Everything Classified as 1

Area Under Curve -- AUC Integrate the Area Under Curve Perfect score is 1 Higher scores allow for generally better tradeoffs Score of 0.5 indicates random model Score of < 0.5 indicates you’re doing something wrong… AUC ~ .89

Operating Points Balance Mistakes for your Application Spam needs low FP Rate Use separate hold out data to find threshold

Pattern for using operating points # Train model and tune parameters on training and validation data # Evaluate model on extra holdout data, reserved for threshold setting xThreshold, yThreshold = ReservedData() # Find threshold that achieves operating point on this extra holdout data potentialThresholds = {} for t in range [ 1% - 100%]: potentialThresholds[t] = FindFPRate(model.Predict(xThreshold, t), yThreshold) bestThreshold = FindClosestThreshold(<target>, potentialThresholds) # Evaluate on test data with selected threshold to estimate generalization performance performanceAtOperatingPoint = FindFNRate(model.Predict(xTest, bestThreshold), yTest) # make sure nothing went crazy… if FindFPRate(model.Predict(xTest, bestThreshold), yTest) <is not close to> potentialThresholds[bestThreshold]: # Problem?