1. Division of Population Health Sciences Royal College of Surgeons in Ireland Coláiste Ríoga na Máinleá in Éirinn Indices of Performances of CPRs Nicola Motterlini, Borislav D Dimitrov, Tom Fahey 1

2. Division of Population Health Sciences Contents CPR derivation –Analytical approach CPR validation –Indices of discrimination –Indices of calibration Performance of a CPR in several validation studies (systematic review with meta-analysis) 2

3. Division of Population Health Sciences Stages of development of a CPR 3

4. Division of Population Health Sciences Multivariable approach (e.g., logistic regression model) Analytical approach to CPR(an example) Target disorder (presence or absence) 4 Prediction LOW RISK % Total score ≤ 3 MODERATE RISK % 4 ≤ Total score ≤ 7 HIGH RISK % Total score ≥ 8 Decision NO TREATMENT FURTHER DIAGNOSTIC TESTING TREATMENT “Weights” ( e.g. odds ratio) 1.9 (1.4-2.5) 5.4 (4.1-7.1) 16.2 (12.2-21.6) 5.7 (4.3-7.5) 1.8 (1.5-2.1) 2.3 (1.8-3.1) 18.2 (14.2-23.6) Points 123123 2 1 2 3 Total score: 3+2+1 = 6 Independent risk factors Age (years) 60-74 75-84 ≥85 Sex male Diabetes mellitus Heart failure Current smoker

5. Division of Population Health Sciences Thresholds approach to diagnosis Diagnosis / test threshold Test / reassurance threshold Probability of disease 100% HIGH RISK 0% LOW RISK Treatment Further diagnostic testing No treatment 5 10 9 8 7 6 5 4 3 2 1 Score

6. Division of Population Health Sciences Stages of development of a CPR 6

7. Division of Population Health Sciences Performance of a CPR can be assessed like a performance of a diagnostic test! TP, number of true positives, FP, number of false positives, FN, number of false negatives, TN, number of true negatives, NT+, the number of patients with positive test results, NT-, the number of patients with negative test results, ND+, the number of patients with the disease, and ND−, number of patients without the disease Disease + Disease - Total Test + score ≥ Cut-off TPFPNT+ Test - score < Cut-off FNTNNT- TotalND+ND-N 7

8. Division of Population Health Sciences Indices of performance 1.For discrimination (classification) –Ability to discriminate between diseased individuals and non-diseased individuals (e.g., sensitivity, specificity, ROC curve, odds ratio, positive and negative predictive values, likelihood ratio etc.) 2.For calibration (comparison) –Agreement between predicted probabilities and observed probabilities (e.g., risk ratio, Hosmer-Lemeshow goodness-of-fit test etc.) 8

9. Division of Population Health Sciences Sensitivity = TP/ND+ Specificity = TN/ND- Disease + Disease - Total Test + score ≥ Cut-off TPFPNT+ Test - score < Cut-off FNTNNT- TotalND+ND-N 1.Discrimination – sensitivity and specificity 9

10. Division of Population Health Sciences Disease risk (%) Predicted Observed Risk ratio -> Low Moderate High Score risk 2.Calibration – risk ratio Predicted Observed <1 “under-prediction” =1 “accurate prediction” >1 “over-prediction” 10

11. Division of Population Health Sciences Performance of a CPR in several validation studies (systematic review with meta-analysis) 11

12. Division of Population Health Sciences Pooled analysis When the 95% CI includes 1 there is no significant statistical difference between the observed and predicted by CPR The smaller the interval, the more accuracy are the results 12

13. Division of Population Health Sciences Index of heterogeneity I 2 value over 50% is a strong indicator for heterogeneity 13

14. Division of Population Health Sciences Summary CPR derivation –Multivariable approach to develop a CPR CPR validation –Indices of discrimination (sensitivity, specificity) –Indices of calibration (risk ratio) Systematic review with meta-analysis of validation studies of CPRs 14