1. Performance Measures II
Prof. Kislaya Prasad BUDT 733 (Spring 2015)

2. The Lift Charts Lift charts analyze the “class of interest”
Fraudulent claims, responders to mailing, patients at risk of a heart attack, … , or customers that prefer regular beer!
Builds on the probability of belonging to the class of interest
Sorts all records in a descending order of probability of belonging to the class
Cumulative number of cases is on the x-axis
Cumulative number of true positives is on the y-axis
College Park, Spring 2015

3. Sort by Probability of Success
Row Id.
Predicted Class
Actual Class
Prob. for 1 (success)
Log odds
Gender
Married
Income
Age
Cumulative Actual Class 99 1
$70,981.00 45 84
$53,768.00 34 2
100
$48,259.00 32 3 59
$49,585.00 4 91
$43,593.00 27 5 74
$42,775.00 6 98
$47,495.00 35 7 60
$44,802.00 8 78
$30,841.00 24 9 79
$52,341.00 42 10 83
$55,281.00 48 11 57
$36,821.00 12 80
$41,779.00 39 13 70
$26,598.00 29 14 86
$63,555.00 62 15 72
$46,626.00 46 16
$38,176.00 40 61
$43,194.00 47 17
$28,513.00 63
$38,671.00 44 18
$23,234.00 31
$44,955.00 53 93
$46,273.00 56 19 28
$28,094.00
$37,263.00 49
$43,741.00 54 50
$26,186.00 38 22
$29,293.00 43
$42,051.00 58
$32,739.00
$33,027.00 51
$22,408.00
$27,078.00
$36,030.00 55 88
$52,875.00 71 20 33
$39,923.00
$24,302.00
$25,440.00 25
$40,582.00 66
$35,678.00 64
Construct cumulative of Actual Class
Plot
20 of 40 cases in validation data are successes. In any M randomly selected cases we will have M/2 successes. This line is denoted in red
Sort by Probability of Success
College Park, Spring 2015

4. Lift Chart (Validation Data)
Lift Charts generated in R (See script for details)
Reference line, represents selecting records (in our case customers) using the Naïve rule

5. ROC (Receiver Operating Characteristic) Curve
Developed in the 1950s for signal detection theory to analyze noisy signals
Characterizes the trade-off between positive hits and false alarms
Performance of each classifier represented as a point on the ROC curve
Changing the threshold of an algorithm, the sample distribution or cost matrix changes the location of the point
Decide where to set the cutoffs to find out false positives and false negatives
College Park, Spring 2015

6. ROC (Receiver Operating Characteristic) Curve
The ROC curve plots the pairs (1-specificity, sensitivity)
Interesting points on the curve:
(0,0): declare everything to be 0
(1,1): declare everything to be 1
(1,0): perfect classification
Diagonal line: Random guessing
Below diagonal line:
prediction is opposite of the true class – the model is worse than random guessing!
Cutoff 0
Cutoff 1
College Park, Spring 2015

7. Creating a ROC curve with R
We have already calculated sensitivity and specificity in our data table example
Add 1-specificity column
Insert plot, with 1-specificity as the x values and sensitivity as the y values
See script for details
College Park, Spring 2015

8. Using ROC for Model Comparison
In this example neither model consistently outperforms the other
M1 is better for small sensitivity values
M2 is better for larger sensitivity values
College Park, Spring 2015

9. Summary
The performance measures differ depending on the use of the model
In data-mining we are usually more interested in prediction accuracy than statistical fit
When predicting numerical values we can use numerous error measures such as the mean absolute error or RMSE.
When classifying we again have many options: accuracy, sensitivity, misclassification costs, the ROC curve etc.
Whenever possible, we should use a validation sample to estimate the performance of the model
If the sample size is too small to split up the model – there are methods to estimate the prediction accuracy
College Park, Spring 2015

10. Next We will introduce two more classification methods!
Naive Bayes – for classification
K-nearest neighbor – for classification and prediction
Both identified as one of the top 10 algorithms by the IEEE International Conference on Data Mining (ICDM) in December 2006! (see here:
College Park, Spring 2015