Model Evaluation Saed Sayad www.ismartsoft.com.

Slides:



Advertisements
Similar presentations
...visualizing classifier performance in R Tobias Sing, Ph.D. (joint work with Oliver Sander) Modeling & Simulation Novartis Pharma AG 3 rd BaselR meeting.
Advertisements

...visualizing classifier performance Tobias Sing Dept. of Modeling & Simulation Novartis Pharma AG Joint work with Oliver Sander (MPI for Informatics,
Kin 304 Regression Linear Regression Least Sum of Squares
Learning Algorithm Evaluation
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.
Chapter 5 – Evaluating Classification & Predictive Performance
Chapter 4 – Evaluating Classification & Predictive Performance © Galit Shmueli and Peter Bruce 2008 Data Mining for Business Intelligence Shmueli, Patel.
Chapter 6: Model Assessment
Assessing and Comparing Classification Algorithms Introduction Resampling and Cross Validation Measuring Error Interval Estimation and Hypothesis Testing.
Linear Regression Demo using PolyAnalyst Generating Linear Regression Formula Generating Regression Rules for Categorical classification.
Model Evaluation Metrics for Performance Evaluation
CS 8751 ML & KDDEvaluating Hypotheses1 Sample error, true error Confidence intervals for observed hypothesis error Estimators Binomial distribution, Normal.
Performance measures Morten Nielsen, CBS, BioCentrum, DTU.
Tutorial 2 LIU Tengfei 2/19/2009. Contents Introduction TP, FP, ROC Precision, recall Confusion matrix Other performance measures Resource.
Statistical Fridays J C Horrow, MD, MSSTAT
CLUSTERING (Segmentation)
ROC & AUC, LIFT ד"ר אבי רוזנפלד.
Notes on Measuring the Performance of a Binary Classifier David Madigan.
Decision Tree Models in Data Mining
CSCI 347 / CS 4206: Data Mining Module 06: Evaluation Topic 07: Cost-Sensitive Measures.
Chapter 8 Introduction to Hypothesis Testing
Copyright © 2006, SAS Institute Inc. All rights reserved. Predictive Modeling Concepts and Algorithms Russ Albright and David Duling SAS Institute.
An Evaluation of A Commercial Data Mining Suite Oracle Data Mining Presented by Emily Davis Supervisor: John Ebden.
Evaluation – next steps
Performance measurement. Must be careful what performance metric we use For example, say we have a NN classifier with 1 output unit, and we code ‘1 =
Data Mining Overview. Lecture Objectives After this lecture, you should be able to: 1.Explain key data mining tasks in your own words. 2.Draw an overview.
Copyright © 2003, SAS Institute Inc. All rights reserved. Cost-Sensitive Classifier Selection Ross Bettinger Analytical Consultant SAS Services.
Sensitivity Sensitivity answers the following question: If a person has a disease, how often will the test be positive (true positive rate)? i.e.: if the.
Allan Mitchell SQL Server MVP Konesans Limited ww.SQLIS.com.
MEASURES OF TEST ACCURACY AND ASSOCIATIONS DR ODIFE, U.B SR, EDM DIVISION.
Regression Regression relationship = trend + scatter
Regression Lines. Today’s Aim: To learn the method for calculating the most accurate Line of Best Fit for a set of data.
Loan Default Model Saed Sayad 1www.ismartsoft.com.
Model Evaluation l Metrics for Performance Evaluation –How to evaluate the performance of a model? l Methods for Performance Evaluation –How to obtain.
Copyright © 2013, 2009, and 2007, Pearson Education, Inc. Chapter 14 Comparing Groups: Analysis of Variance Methods Section 14.3 Two-Way ANOVA.
Preventing Overfitting Problem: We don’t want to these algorithms to fit to ``noise’’ Reduced-error pruning : –breaks the samples into a training set and.
Copyright © 2012 Pearson Education, Inc. All rights reserved Chapter 12 Multiple Regression and Model Building.
Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 5: Credibility: Evaluating What’s Been Learned.
Copyright © 2003, N. Ahbel Residuals. Copyright © 2003, N. Ahbel Predicted Actual Actual – Predicted = Error Source:
Copyright © 2013, 2009, and 2007, Pearson Education, Inc. Chapter 3 Association: Contingency, Correlation, and Regression Section 3.3 Predicting the Outcome.
1 Performance Measures for Machine Learning. 2 Performance Measures Accuracy Weighted (Cost-Sensitive) Accuracy Lift Precision/Recall –F –Break Even Point.
MODELING RESPONSE TO DIRECT MAIL MARKETING ISQS 7342 – Dr. Zhangxi Lin by Junil Chang.
Evaluating Classification Performance
Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.
Logistic Regression Saed Sayad 1www.ismartsoft.com.
Chapter 5: Credibility. Introduction Performance on the training set is not a good indicator of performance on an independent set. We need to predict.
Evaluating Classifiers Reading: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)An introduction to ROC analysis.
Chapter 5 – Evaluating Predictive Performance Data Mining for Business Analytics Shmueli, Patel & Bruce.
Copyright © 2003, N. Ahbel Residuals. Copyright © 2003, N. Ahbel Predicted Actual Actual – Predicted = Error Source:
Chapter 5: Credibility. Introduction Performance on the training set is not a good indicator of performance on an independent set. We need to predict.
Data Analytics CMIS Short Course part II Day 1 Part 4: ROC Curves Sam Buttrey December 2015.
Performance measures Morten Nielsen, CBS, Department of Systems Biology, DTU.
Timothy Wiemken, PhD MPH Assistant Professor Division of Infectious Diseases Diagnostic Tests.
2011 Data Mining Industrial & Information Systems Engineering Pilsung Kang Industrial & Information Systems Engineering Seoul National University of Science.
Evaluation – next steps
Numeracy Resources for KS1
Kin 304 Regression Linear Regression Least Sum of Squares
Performance Measures II
Data Mining Classification: Alternative Techniques
Features & Decision regions
Regression Computer Print Out
Case Study 5 An Investigation of Factors Affecting the Sale Price of Condominium Units Sold at Public Auction.
Evaluation and Its Methods
Evaluating Classifiers (& other algorithms)
Figure 1. Table for calculating the accuracy of a diagnostic test.
Evaluation and Its Methods
Evaluating Classifiers
—ROC curves for each simple test compared with NCS (gold standard) plotting the sensitivity versus 1-specificity (the false-positive rate) for different.
RANDOM NUMBERS SET # 1:
Presentation transcript:

Model Evaluation Saed Sayad www.ismartsoft.com

Data Mining Steps 1 2 3 4 5 6 www.ismartsoft.com Problem Definition Data Preparation 3 Data Exploration 4 Modeling 5 Evaluation 6 Deployment www.ismartsoft.com

Model Evaluation www.ismartsoft.com Evaluation Classification Confusion Matrix Gain, Lift, ... Charts Regression Mean Squared Error Residuals Chart www.ismartsoft.com

Classification - Confusion Matrix Positive Cases Negative Cases CM True Positive False Negative Predicted Positive Predicted Negative www.ismartsoft.com

Confusion Matrix - Evaluation Measurements Actual + - TP FP TP+FP FN TN FN+TN TP+FN FP+TN TP+FP+FN+TN Predicted

Sensitivity and Specificity www.ismartsoft.com

Classification – Gain Chart Target% Wizard 100% Model Random Population% 0% 50% 100% www.ismartsoft.com

Gain Chart Wizard 100% A 50% Random 10% Target% Population% 10% 18% www.ismartsoft.com

Gain Chart Score Table Sorted by Score Gain Table Target Score 235 1 235 1 724 556 345 480 676 195 880 368 ... Target Score 1 880 724 676 556 480 368 345 235 195 ... Count% Target% 10 36 20 54 30 66 40 76 50 85 60 90 70 94 80 98 100 www.ismartsoft.com

Classification – Gain Chart Target% 100% A 85% 76% B 66% 54% 36% Population% 10% 20% 30% 40% 50% 100% Copyright iSmartsoft Inc. 2008 www.ismartsoft.com

Lift Chart Gain Table Lift Table Count% Target% 10 36 20 54 30 66 40 76 50 85 60 90 70 94 80 98 100 Count% Lift 10 3.6 20 2.7 30 2.2 40 1.9 50 1.7 60 1.5 70 1.3 80 1.2 90 1.1 100 1 Copyright iSmartsoft Inc. 2008 www.ismartsoft.com

Lift Chart Lift Population% www.ismartsoft.com

K-S Chart (Kolmogorov-Smirnov) Score Range Count Cumulative Count Lower Upper Target Non-Target K-S 100 3 62 0.5% 0.8% 0.3% 200 23 1.1% 0.6% 300 1 66 0.7% 2.0% 1.3% 400 7 434 7.7% 5.7% 500 181 5627 34.3% 81.7% 47.4% 600 112 886 54.3% 93.3% 39.0% 700 83 332 69.1% 97.7% 28.6% 800 45 63 77.1% 98.5% 21.4% 900 29 37 82.3% 99.0% 16.7% 1000 99 77 100.0% 0.0% K-S K(0.95) = 6.0%    K(0.99) = 7.1%  www.ismartsoft.com

K-S Chart Count% Score www.ismartsoft.com

ROC Chart (Receiver Operating Characteristic) Count% False Positive Rate (1-Specificity) True Positive Rate (Sensitivity) 10 0.1 0.66 20 0.2 0.79 30 0.3 0.86 40 0.4 0.91 50 0.5 0.94 60 0.6 0.95 70 0.7 0.98 80 0.8 90 0.9 0.99 100 1.0 1.00 www.ismartsoft.com

ROC Chart Sensitivity 1-Specificity www.ismartsoft.com

Regression – Mean Squared Error www.ismartsoft.com

Regression – Relative Squared Error www.ismartsoft.com

Regression – Mean Absolute Error www.ismartsoft.com

Regression – Relative Absolute Error www.ismartsoft.com

Regression – Standardized Residuals Plot www.ismartsoft.com

Questions? www.ismartsoft.com

Feedback: Privacy Policy Feedback
Do Not Sell My Personal Information
About project: SlidePlayer Terms of Service

© 2025 SlidePlayer.com. Inc. All rights reserved.