Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.

Slides:

Advertisements

Similar presentations

Classification.. continued. Prediction and Classification Last week we discussed the classification problem.. – Used the Naïve Bayes Method Today..we.

Advertisements

...visualizing classifier performance in R Tobias Sing, Ph.D. (joint work with Oliver Sander) Modeling & Simulation Novartis Pharma AG 3 rd BaselR meeting.

...visualizing classifier performance Tobias Sing Dept. of Modeling & Simulation Novartis Pharma AG Joint work with Oliver Sander (MPI for Informatics,

Learning Algorithm Evaluation

Evaluation of segmentation. Example Reference standard & segmentation.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.

Precision and Recall.

The University of Texas at Austin, CS 395T, Spring 2008, Prof. William H. Press 1 Computational Statistics with Application to Bioinformatics Prof. William.

Assessing and Comparing Classification Algorithms Introduction Resampling and Cross Validation Measuring Error Interval Estimation and Hypothesis Testing.

Model Evaluation Metrics for Performance Evaluation

CS 8751 ML & KDDEvaluating Hypotheses1 Sample error, true error Confidence intervals for observed hypothesis error Estimators Binomial distribution, Normal.

Performance measures Morten Nielsen, CBS, BioCentrum, DTU.

Tutorial 2 LIU Tengfei 2/19/2009. Contents Introduction TP, FP, ROC Precision, recall Confusion matrix Other performance measures Resource.

True/False. False True Subject May Go Here True / False ? Type correct answer here. Type incorrect answer here.

Determine whether each curve below is the graph of a function of x. Select all answers that are graphs of functions of x:

ROC Curve and Classification Matrix for Binary Choice Professor Thomas B. Fomby Department of Economics SMU Dallas, TX February, 2015.

Notes on Measuring the Performance of a Binary Classifier David Madigan.

R OBERTO B ATTITI, M AURO B RUNATO. The LION Way: Machine Learning plus Intelligent Optimization. LIONlab, University of Trento, Italy, Feb 2014.

CSCI 347 / CS 4206: Data Mining Module 06: Evaluation Topic 07: Cost-Sensitive Measures.

Evaluating Classifiers

SVMLight SVMLight is an implementation of Support Vector Machine (SVM) in C. Download source from :

People Fractions. Problem 1 of 20 Answer 1 = 10 Problem 2 of 20 Answer 2 = 5.

1 Evaluating Model Performance Lantz Ch 10 Wk 5, Part 2 Right – Graphing is often used to evaluate results from different variations of an algorithm. Depending.

Error estimation Data Mining II Year Lluís Belanche Alfredo Vellido.

Performance measurement. Must be careful what performance metric we use For example, say we have a NN classifier with 1 output unit, and we code ‘1 =

הערכת טיב המודל F-Measure, Kappa, Costs, MetaCost ד " ר אבי רוזנפלד.

Evaluating Hypotheses Reading: Coursepack: Learning From Examples, Section 4 (pp )

Data Mining: Classification & Predication Hosam Al-Samarraie, PhD. Centre for Instructional Technology & Multimedia Universiti Sains Malaysia.

Classification Performance Evaluation. How do you know that you have a good classifier? Is a feature contributing to overall performance? Is classifier.

Machine learning system design Prioritizing what to work on

Computational Intelligence: Methods and Applications Lecture 16 Model evaluation and ROC Włodzisław Duch Dept. of Informatics, UMK Google: W Duch.

Model Evaluation l Metrics for Performance Evaluation –How to evaluate the performance of a model? l Methods for Performance Evaluation –How to obtain.

Preventing Overfitting Problem: We don’t want to these algorithms to fit to ``noise’’ Reduced-error pruning : –breaks the samples into a training set and.

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 5: Credibility: Evaluating What’s Been Learned.

Machine Learning Tutorial-2. Recall, Precision, F-measure, Accuracy Ch. 5.

1 Performance Measures for Machine Learning. 2 Performance Measures Accuracy Weighted (Cost-Sensitive) Accuracy Lift Precision/Recall –F –Break Even Point.

Evaluating Classification Performance

Evaluating Classifiers Reading: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)An introduction to ROC analysis.

Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson.

ROC curve estimation. Index Introduction to ROC ROC curve Area under ROC curve Visualization using ROC curve.

Chapter 5: Credibility. Introduction Performance on the training set is not a good indicator of performance on an independent set. We need to predict.

Data Analytics CMIS Short Course part II Day 1 Part 4: ROC Curves Sam Buttrey December 2015.

Evaluating Classifiers. Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)

Announcements 1. Textbook will be on reserve at library 2. Topic schedule change; modified reading assignment: This week: Linear discrimination, evaluating.

Information Organization: Evaluation of Classification Performance.

Knowledge and Information Retrieval Dr Nicholas Gibbins 32/4037.

Chapter 1: Classification Use Parade of Life: Monerans, Protists, Fungi, and Plants Answer Questions (use complete sentences) –Page 18, #1-3 –Page 25,

Evaluating Classifiers

Machine Learning: Methodology Chapter

Date of download: 11/16/2017 Copyright © ASME. All rights reserved.

Performance Measures II

Data Mining Classification: Alternative Techniques

Features & Decision regions

Advanced Analytics. Advanced Analytics What is Machine Learning?

Evaluating Classifiers (& other algorithms)

Image Classification via Attribute Detection

Learning Algorithm Evaluation

True or False: {image} is one-to-one function.

Model Evaluation and Selection

Evaluating Models Part 1

Computational Intelligence: Methods and Applications

Dr. Sampath Jayarathna Cal Poly Pomona

Evaluating Classifiers

Assignment 1: Classification by K Nearest Neighbors (KNN) technique

CS639: Data Management for Data Science

Dr. Sampath Jayarathna Cal Poly Pomona

Precision and Recall.

True or False True or False

Information Organization: Evaluation of Classification Performance

Presentation transcript:

Quiz 1 review

Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link on class website

“Confusion matrix” for a given class c ActualPredicted (or “classified”) True False (in class c)(not in class c) True (in class c) TruePositiveFalseNegative False (not in class c)FalsePositiveTrueNegative Evaluating classification algorithms

Accuracy: Fraction of correct answers out of all problems Precision: Fraction of true positives out of all predicted positives: Recall: Fraction of true positives out of all actual positives:

Trading off precision against recall w1w1 w2w2 w 64 o w0w0 +1 x1x1 x2x2 x 64 How can we improve precision (at the expense of recall) with a fixed classifier?

True False (“8”)(“0”) True (“8”) 4010 False (“0”)30120 True False (“8”)(“0”) True (“8”) ?? False (“0”)?? Old, with threshold of 0 New, with threshold of: -∞ Example 1: Assume 200 sample digits, of which 50 have class “8” Precision? Recall? Actual Predicted

True False (“8”)(“0”) True (“8”) 4010 False (“0”)30120 True False (“8”)(“0”) True (“8”) ?? False (“0”)?? Old, with threshold of 0 New, with threshold of: +∞ Precision? Recall? Actual Predicted Example 2: Assume 200 sample digits, of which 50 have class “8”

Results of classifier ThresholdAccuracyPrecisionRecall ∞ Creating a Precision/Recall Curve

9 (“sensitivity”) (1  “specificity”)

10

Results of classifier ThresholdAccuracyTPRFPR ∞ Creating a ROC Curve

Precision/Recall versus ROC curves 12

13

14

15