הערכת טיב המודל F-Measure, Kappa, Costs, MetaCost ד " ר אבי רוזנפלד.

Slides:

Advertisements

Similar presentations

...visualizing classifier performance Tobias Sing Dept. of Modeling & Simulation Novartis Pharma AG Joint work with Oliver Sander (MPI for Informatics,

Advertisements

Class Imbalance vs. Cost-Sensitive Learning

Authorship Verification Authorship Identification Authorship Attribution Stylometry.

Evaluation of segmentation. Example Reference standard & segmentation.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.

Cross Validation False Negatives / Negatives

The University of Texas at Austin, CS 395T, Spring 2008, Prof. William H. Press 1 Computational Statistics with Application to Bioinformatics Prof. William.

More Classifier and Accuracy Measure of Classifiers

SUPPORT VECTOR MACHINES PRESENTED BY MUTHAPPA. Introduction Support Vector Machines(SVMs) are supervised learning models with associated learning algorithms.

Assessing and Comparing Classification Algorithms Introduction Resampling and Cross Validation Measuring Error Interval Estimation and Hypothesis Testing.

Model Evaluation Metrics for Performance Evaluation

Classification by Machine Learning Approaches - Exercise Solution Michael J. Kerner – Center for Biological Sequence.

© Vipin Kumar CSci 8980 Fall CSci 8980: Data Mining (Fall 2002) Vipin Kumar Army High Performance Computing Research Center Department of Computer.

1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.

Tutorial 2 LIU Tengfei 2/19/2009. Contents Introduction TP, FP, ROC Precision, recall Confusion matrix Other performance measures Resource.

Jeremy Wyatt Thanks to Gavin Brown

ROC & AUC, LIFT ד"ר אבי רוזנפלד.

Data Mining – Credibility: Evaluating What’s Been Learned

CSCI 347 / CS 4206: Data Mining Module 06: Evaluation Topic 07: Cost-Sensitive Measures.

Classification II (continued) Model Evaluation

Evaluating Classifiers

Chapter 4 Pattern Recognition Concepts continued.

A Study of the Behavior of Several Methods for Balancing Machine Learning Training Data Author: Gustavo E. A. Batista Presenter: Hui Li University of Ottawa.

Evaluation – next steps

1 Evaluating Model Performance Lantz Ch 10 Wk 5, Part 2 Right – Graphing is often used to evaluate results from different variations of an algorithm. Depending.

WEKA and Machine Learning Algorithms. Algorithm Types Classification (supervised) Given -> A set of classified examples “instances” Produce -> A way of.

Performance measurement. Must be careful what performance metric we use For example, say we have a NN classifier with 1 output unit, and we code ‘1 =

Evaluating Hypotheses Reading: Coursepack: Learning From Examples, Section 4 (pp )

Sensitivity Sensitivity answers the following question: If a person has a disease, how often will the test be positive (true positive rate)? i.e.: if the.

Evaluating What’s Been Learned. Cross-Validation Foundation is a simple idea – “ holdout ” – holds out a certain amount for testing and uses rest for.

Classification Performance Evaluation. How do you know that you have a good classifier? Is a feature contributing to overall performance? Is classifier.

Regression Lines. Today’s Aim: To learn the method for calculating the most accurate Line of Best Fit for a set of data.

Computational Intelligence: Methods and Applications Lecture 16 Model evaluation and ROC Włodzisław Duch Dept. of Informatics, UMK Google: W Duch.

Model Evaluation l Metrics for Performance Evaluation –How to evaluate the performance of a model? l Methods for Performance Evaluation –How to obtain.

Bab /57 Bab 4 Classification: Basic Concepts, Decision Trees & Model Evaluation Part 2 Model Overfitting & Classifier Evaluation.

Practical Issues of Classification Underfitting and Overfitting –Training errors –Generalization (test) errors Missing Values Costs of Classification.

N. Gagunashvili (UNAK & MPIK) Methods of multivariate analysis for imbalance data problem Under- and Oversampling Techniques Nikolai Gagunashvili (UNAK.

Costs and Filters Dr. Avi Rosenfeld Department of Industrial Engineering Jerusalem College of Technology

Class Imbalance Classification Implementation Group 4 WEI Lili, ZENG Gaoxiong,

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 5: Credibility: Evaluating What’s Been Learned.

Machine Learning Tutorial-2. Recall, Precision, F-measure, Accuracy Ch. 5.

Class Imbalance in Text Classification

1 Performance Measures for Machine Learning. 2 Performance Measures Accuracy Weighted (Cost-Sensitive) Accuracy Lift Precision/Recall –F –Break Even Point.

Evaluating Classification Performance

Bayesian decision theory: A framework for making decisions when uncertainty exit 1 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e.

Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Genetic Algorithms (in 1 Slide) l GA: based on an analogy to biological evolution l Each.

Chapter 5: Credibility. Introduction Performance on the training set is not a good indicator of performance on an independent set. We need to predict.

Evaluating Classifiers Reading: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)An introduction to ROC analysis.

CS 2750: Machine Learning The Bias-Variance Tradeoff Prof. Adriana Kovashka University of Pittsburgh January 13, 2016.

Intelligent Database Systems Lab 國立雲林科技大學 National Yunlin University of Science and Technology Advisor ： Dr. Hsu Presenter ： Chien-Shing Chen Author: Gustavo.

Chapter 5: Credibility. Introduction Performance on the training set is not a good indicator of performance on an independent set. We need to predict.

Classification Cheng Lei Department of Electrical and Computer Engineering University of Victoria April 24, 2015.

Great Workshop La Palma -June 2011 Handling Imbalanced Datasets in Multistage Classification Mauro López Centro de Astrobiología.

Evolving Decision Rules (EDR)

Balancing Techniques Gretel Fernández.

Performance Evaluation 02/15/17

CSE 4705 Artificial Intelligence

Performance Measures II

Chapter 7 – K-Nearest-Neighbor

Data Mining Classification: Alternative Techniques

Features & Decision regions

Evaluating Models Part 1

Computational Intelligence: Methods and Applications

Data Mining Class Imbalance

Dr. Sampath Jayarathna Cal Poly Pomona

Assignment 1: Classification by K Nearest Neighbors (KNN) technique

Dr. Sampath Jayarathna Cal Poly Pomona

Presentation transcript:

הערכת טיב המודל F-Measure, Kappa, Costs, MetaCost ד " ר אבי רוזנפלד

הגדרות נוסחהמשמעותמדד TP / (TP + FP) בכמה אחוזים דייקתי בקטגוריה מסוימת ( מתוך אלו שאמרתי הם בקטגוריה ) Precision TP / (TP + FN) בכמה אחוזים דייקתי בכלל במופעים בקטגוריה Recall 2*precision*recall/ precision+recall Harmonic mean of precision and recall F-Measure (TP + TN) / ( הכול ) הדיוק הכללי Accuracy

False Positives / Negatives PN P2010 N3090 Predicted Actual Confusion matrix 1 PN P1020 N15105 Predicted Actual Confusion matrix 2 FN FP Precision (P) = 20 / 50 = 0.4 Recall (P) = 20 / 30 = F-measure=2*.4*.666/1.0666=.5

The Kappa Statistic Kappa measures relative improvement over random prediction D real / D perfect = A (accuracy of the real model) D random / D perfect = C (accuracy of a random model) Kappa Statistic = (A-C) / (1-C) = (D real / D perfect – D random / D perfect ) / (1 – D random / D perfect ) Remove D perfect from all places (D real – D random ) / (D perfect – D random ) Kappa = 1 when A = 1 Kappa  0 if prediction is no better than random guessing

Aside: the Kappa statistic Two confusion matrix for a 3-class problem: real model (left) vs random model (right) Number of successes: sum of values in diagonal (D) Kappa = (D real – D random ) / (D perfect – D random ) – (140 – 82) / (200 – 82) = – Accuracy = 140/200 = 0.70 abc a b c Actual Predicted total abc a b c Actual Predicted total

The kappa statistic – how to calculate D random ? abc a b c Actual total abc a?100 b60 c Actual total 100*120/200 = 60 Rationale: 100 actual values, 120/200 in the predicted class, so random is: 100*120/200 Actual confusion matrix, C Expected confusion matrix, E, for a random model

Cost-Sensitive Learning מוסיפים עלות לכל סוג של סיווג בדרך כלל, אלגוריתמים לא מפרידים בין קטגוריות אבל יש ה - ר - ב - ה יישומים שזה חשוב : –אבחון סרטן –אבחון סיכונים ( התקפות מחשב, זיוף, וכו ')

Class Imbalance vs. Asymmetric Misclassification costs Class Imbalance: one class occurs much more often than the other Asymmetric misclassification costs: the cost of misclassifying an example from one class is much larger than the cost of misclassifying an example from the other class. לפי הניסיון שלי : שני המוסגים בדרך כלל באים ביחד גם הפתרונות דומות דוגמאות : –תוריד שורות מה MAJORITY CLASS –תוסיף שורות ל MINORITY CLASS –תוסיף עלות ל MINORITY -- METACOST

Making Classifier Balanced Baseline Methods – Random over-sampling – Random under-sampling Under-sampling Methods – Tomek links – Condensed Nearest Neighbor Rule – One-sided selection – CNN + Tomek links – Neighborhood Cleaning Rule Over-sampling Methods – Smote Combination of Over-sampling method with Under-sampling method – Smote + Tomek links – Smote + ENN

איזה מצב יותר טוב ??? PN P2010 N3090 Predicted Actual PN P02 N10 Confusion matrix 2 Cost matrix PN P1020 N15105 Predicted Actual Confusion matrix 1 FN FP Error rate: 40/150 Cost: 30x1+10x2=50 Error rate: 35/150 Cost: 15x1+20x2=55 FN

MetaCost By wrapping a cost-minimizing procedure, “meta- learning” stage, around the classifier תתייחס לסוג הלמידה כ " קופצה שחורה " אפשר לשנות את ה COST ולהשפיע על התוצאות

דוגמא : מתי אנשים מפעילים CRUISE CONTROL

מתוך WEKA Metacost בלי תוספת עלות

מתוך WEKA Metacost עם עלות