1. A SAS Macro to Compute Added Predictive Ability of New Markers in Logistic Regression
Kevin Kennedy, MS
Saint Luke’s Hospital, Kansas City, MO
Kansas City Area SAS User Group Meeting
September 3, 2009

2. Acknowledgment
A special thanks to Michael Pencina (PhD, Biostatistics Dept, Boston University, Harvard Clinical Research Institute) for his valuable input, ideas, and additional formulas to strengthen output.

3. Motivation Predicting dichotomous outcomes are important
Will a patient develop a disease?
Will an applicant default on a loan?
Will KSU win a game in the NCAA tournament?
Improving an already usable models should be a continued goal

4. Motivation Many published models exist Risk of Bleeding after PCI
Predicts patient risk of bleeding after a PCI (Percutaneous Coronary Intervention) based on 10 patient characteristics: Age, gender, shock, prior intervention, kidney function, etc..
Example: a 78 year old female with Prior Shock and Congestive Heart Failure has a ~8% chance of bleeding after procedure (national avg=2-3%)
Why important? We can treat those at high risk appropriately
Mehta et al. Circ Intervention, June 2009

5. However….. These models aren’t etched in stone
New markers (variables) should be investigated to improve model performance
Important Question: How do we determine if we should add these new markers to the model?

6. Project Goal
Compare:
How Much Does the new variable add to model performance?
Output of Interest: Predicted Probability of Event (one for each model) computed with the logistic regression model
Talk about a probability for each model

7. Outline Traditional Comparisions New Measures
Receiver Operating Characteristic Curve
New Measures
IDI, NRI, Vickers Decision Curve
SAS macro to obtain output

8. Traditional Approach-AUCs
Common to plot the Receiver Operating Characteristic Curve (ROC) and report the area underneath (AUC) or c-statistic
Measures model discrimination
Equivalent to the probability that the predicted risk is higher for an event than a non-even[4]

9. AUC/ROC Background
ROC/AUC plot depicts trade-off between benefit (true positives) and costs (false positives) by defining a “cut off” to determine positive and negative individuals
Cut Off
Events
Non-Events
True Neg
True Pos
False Neg
False Pos
Predicted Probabilities
True Positive Rate (Sensitivity)=.9
False Positive Rate (1-Specificity)=.4

10. ROC curve True Positive Rate (sensitivity)0%
100%
False Positive Rate (1-specificity) 0%
100%

11. AUC Computation
All possible pairs are made between events and nonevents
A dataset with 100 events and 1000 non-events would have 100*1000= pairs
If the predicted probability for the event is higher for the subject actually experiencing the event give a ‘1’ (concordant) otherwise ‘0’ (discordant)
C-statistic is the average of the 1’s and 0’s (.5 for ties)
Now use methods by DeLong to compare AUCs of Model1 and Model2

12. Advantages Used all the time
Recommended guidelines exist for Excellent/Good/Poor discrimination
Default output in proc logistic, with new extensions in version 9.2 (roc and roccontrast statements) along with other computing packages

13. Disadvantages Rank based Doesn’t imply a useful model
A comparison of .51 to .50 is treated the same as .7 to .1
Doesn’t imply a useful model
Example: all events have probability=.51 and non-event have probability=.5---Perfect discrimination (c=1) but not useful
Extremely hard to find markers that result in high AUC
Pepe [5] claims an odds ratio of 3 doesn’t yield good discrimination

14. Alternatives
Pencina and D’Agostino in 2008 (Statistics in Medicine) suggest 2 additional statistics:
IDI (Integrated Discrimination Improvement)
NRI (Net Reclassification Improvement)
Vickers [2,3] developed graphical techniques of comparing models

15. IDI Measure of improved sensitivity without sacrificing specificity
Formula measures how much increase in ‘p’ for events, and how much decrease for the non-event

16. IDI Absolute_IDI=(.15-.12)+(.25-.2)=.08 160% Relative improvement
Mean ‘p’ increased by 5% for events
Mean ‘p’ decreased by 3% for non-events

17. NRI Measures how well the new model reclassifies event and non-event
Dependant on how we decide to classify observations
Example: 0-10% Low, 10-20% Moderate, >20% High
Questions: do the patients experiencing an event go up in risk? Mod to High between model1 and 2
Do Patients not experiencing an event go down in risk? Moderate to Low?

18. NRI
Formula:

19. NRI Computation Example
3 groups defined
<10% (low)
10-20% (moderate)
>20% (high)
Each individual will have a group for model 1 and 2
2 cross-tabulation tables (events and non-events)

20. NRI Computation Example
Crosstab1: Events (100 Events)
Model 2
Low
Mod
High 10 8 2 3 30 5
20 Events moving up
10 Events moving down
70 Events not moving
Net of 10/100 (10%) of events getting reclassified correctly
Model 1

21. NRI Computation Example
Crosstab2: Non-Events (200 Non-Events)
Model 2
15 Non-events moving up
35 Non-events moving down
150 Non-events not moving
Net of 20/200 (10%) of non-events getting reclassified correctly
Low
Mod
High 50 5 20 40 10 60
Model 1

22. NRI Caveats Dependant on Groups Alternative ways to define groups
Would we reach similar conclusions with groups: <15, 15-30, >30?????
Alternative ways to define groups
Any up/down movement. A change in p from .15 to .151 would be an ‘up’ movement
A threshold: ex. a change of 3% would constitute an up/down. i.e. .33 to .37 would be an ‘up’ but .33 to .34 would be no movement
Good News: Macro handles all these cases and you can request all at once

23. Vickers Decision Curve
A graphical comparison of model 1 and 2 based off of ‘net’ benefit (first attributed to Peirce-1884)
Useful if a threshold is important.
Example: If a persons predicted probability of an outcome is greater than 10% we treat with strategy A
Here we’d want to compare the models at this threshold

24. Vickers Decision Curve
Example:
N=1000, dichotomous event, 10% as threshold
True Positive count
Outcome
Yes No
≥10% 80
200
<10% 20
700
False Positive count
Predicted Probability
5.78 net true positive results per 100 compared with treating all as negative

25. Vickers Decision Curve
No difference between 1 2 and treat all
0.06
0.05
0.04
Model 2 seems to outperform 1
0.03
Net Benefit
0.02
0.01
0.00
-0.01 10 20 30 40 50
Threshold Probability in %
PLOT
Treat All
Model1
Model2

26. SAS Macro to Obtain Output
%added_pred(data= ,id= , y= , model1cov= ,model2cov= , nripoints=ALL, nriallcut=%str(), vickersplot=FALSE, vickerpoints=%str());
Model1cov=initial model
Model2cov=new model
Nripoints=nri levels (eg: <10, 10-20, >20) insert nripoints=.1 .2
Nriallcut=if you want to test amount of increase or decrease instead of levels, i.e. if you want to know if a person increase/decreases by 5% insert nriallcut=.05
Vickerpoints=thresholds to test (eg, 10%)

27. AUC section of Macro AUC Comparisions are easy with SAS 9.2
PROC LOGISTIC DATA=&DATA;
MODEL &y=&model1cov &model2cov;
ROC ‘FIRST’ &model1cov;
ROC ‘SECOND’ &model2cov;
ROCCONTRAST REFERENCE=‘FIRST’;
ODS OUTPUT ROCASSOCIATION=ROCASS ROCCONTRASTESTIMATE=ROCDIFF;
RUN;
If working from an earlier version the %roc macro will be called from the SAS website[6]

28. AUC Section of Macro Sample output
%added_pred(data=data, id=id, y=event, model1cov=x1 x2 x3 x4, model2cov=x1 x2 x3 x4 x5, …….);
Model1 AUC
Model2 AUC
Difference in AUC
Std Error for Difference
P-value for difference
95% CI for Difference
0.0147
.0042
0.0005
( ,0.0229)

29. IDI Section of Macro
Use proc logistic output dataset for model1 and model2
PROC LOGISTIC DATA=&DATA;
MODEL &y=&model1cov;
OUTPUT OUT=OLD PRED=P_OLD;
RUN;
MODEL &y=&model2cov;
OUTPUT OUT=NEW PRED=P_NEW;
proc sql noprint;
create table probs as select *, (p_new-p_old) as pdiff
from old(keep=&id &y &model1cov &model2cov p_old ) as a join new(keep=&id &y &model1cov &model2cov p_new ) as b on a.&id=b.&id
order by &y;
quit;
Now use proc means or sql to obtain: p_event_new, p_event_old, p_nonevent_new, p_nonevent_old

30. IDI Section of Macro Sample Output IDI IDI Std Err Z-value P-value
95% CI
Probability change for events
Probability change for non-events
Relative IDI
.0207
.0064
6.3
<.0001
(0.0143, )
.0186
.167

31. NRI Section of Macro In 3 parts:
All groups (any up/down movement)
User defined (eg. <10,10-20,>20)
Threshold (eg. a change of 3%)
Coding is more involved here containing
Counts for number of groups
Do-loops for various # of thresholds and user groups

32. NRI Section of Macro User Defined Groups(eg
NRI Section of Macro User Defined Groups(eg. <10, 10-20,>20 nripoints=.1 .2)
%if &nripoints^=ALL %then %do;
%let numgroups=%eval(%words(&nripoints)+1); /*figure out how many ordinal groups*/
data nriprobs;
set probs;
/*define first ordinal group for pre and post*/
if 0<=p_old<=%scan(&nripoints,1,' ') then group_pre=1;
if 0<=p_new<=%scan(&nripoints,1,' ') then group_post=1;
%let i=1;
%do %until(&i>%eval(&numgroups-1));
if %scan(&nripoints,&i,' ')<p_old then do;group_pre=&i+1;end;
if %scan(&nripoints,&i,' ')<p_new then do;group_post=&i+1;end;
%let i=%eval(&i+1);
%end;
if &y=0 then do;
if group_post>group_pre then up_nonevent=1;
if group_post<group_pre then down_nonevent=1;
end;
if &y=1 then do;
if group_post>group_pre then up_event=1;
if group_post<group_pre then down_event=1
if up_nonevent=. then up_nonevent=0;
if down_nonevent=. then down_nonevent=0;
if up_event=. then up_event=0;
if down_event=. then down_event=0;
run;

33. NRI Section of Macro Sample Output
%added_pred(data=,…..,nripoints=.1 .2, nriallcut=.03,…..);
Group
NRI
STD ERR
Z-value
P-VALUE
95% CI
% of events correctly reclassified
% of non-event correctly reclassified
ALL
.454
.09
9.7
<.0001
(0.3649,0.5447)
(10%)
56%
CUT_.03
.101
.08
2.46
.014
(0.0205,0.1817) 4% 6%
USER
.127
.05
4.95
(0.0769,0.177) 5% 8%

34. Vickers’ Section of Macro
Default is no analysis
Can test multiple thresholds
Uses bootstrapping techniques to create 95% CI’s
If testing thresholds run time will increase due to Bootstrapping

35. Vickers Section of Macro
Code makes 101 datasets: vicker&i testing at every integer threshold (0-100) to make graph
%if &vickersplot=TRUE %then %do;
%do i=0 %to 100;
data vicker&i;
set probs(keep=&id p_new p_old &y);
if p_new>&i/100 then predmodel=1;
else predmodel=0;
if p_old>&i/100 then predmodel_old=1;
else predmodel_old=0;
run;
Now use proc freq to obtain frequency counts (see example)
Set all the datasets and delete individual parts:
data vicker;
set %do i=0 %to 100; vicker&i %end;;
proc datasets library=work nolist;
delete %do i=0 %to 100; vicker&i %end;;
quit;

36. Vickers Section of Macro
%added_pred(data=,…..,vickersplot=TRUE,…..)

37. Vickers Section-Threshold Analysis
Can test multiple thresholds, i.e. 15% and 20% by use of bootstrapping methods
If you know a priori that if an individuals risk is greater then some threshold (t) you’d treat differently, so you’d want to compare models at this point

38. Vickers Section-Threshold Analysis
Sample Output
%added_pred(data=, ….., vickersplot=TRUE, vickerpoints=.15);
Cut-point
Mean(CI) for diff in net benefit (new-old)
Mean(CI) for net benefit of old model
Mean (CI) for diff in net benefit (old-treatall)
Mean(CI) for net benefit of new model
Mean (CI) for diff in Net benefit (new-treatall) 15
( ,0.005)
0.0295
(0.0229,0.362)
0.0852
(0.0783,0.0922)
0.0308
(0.0244,0.0374)
0.0866
(0.0799,0.093)

39. Conclusions
Don’t rely only on AUC and statistical significance to access added marker predictive ability, but a combination of methods
Future: extend to time-to-event analysis

40. Q’s or Comments
If you want to use the macro or obtain literature contact me at: or

41. References
1) Pencina MJ, D'Agostino RB Sr, D’Agostino RB Jr. Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Stat Med 2008; 27:
2) Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making Nov-Dec;26(6):565-74
3) Vickers AJ, Cronin AM, Elkin EB, Gonen M. Extensions to decision curve analysis, A novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Medical Informatics and
Decision Making Nov 26;8(1):53.
4) Cook NR. Use and Misuse of the receiver operating characteristics curve in risk prediction. Circulation 2007; 115:
5) Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol. 2004;159: 6)

42. %words macro %macro words(list); %local count; %let count=0;
%do %while(%qscan(&list,&count+1,%str( )) ne %str());
%let count=%eval(&count+1);
%end;
&count
%mend words;