Presentation is loading. Please wait.

Presentation is loading. Please wait.

© 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 1 Evaluation of Support Vector Machines for Risk Modeling in Interventional.

Published byCleopatra Eaton Modified over 9 years ago

Similar presentations

Presentation on theme: "© 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 1 Evaluation of Support Vector Machines for Risk Modeling in Interventional."— Presentation transcript:

1 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 1 Evaluation of Support Vector Machines for Risk Modeling in Interventional Cardiology Michael E. Matheny, M.D.

2 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 2 Goal Comparison of support vector machines and logistic regression risk modeling performance over time for the outcome of death in pre- intervention cardiac catheterization patients. Comparison of support vector machines and logistic regression risk modeling performance over time for the outcome of death in pre- intervention cardiac catheterization patients.

3 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 3 Pre-intervention Risk Assessment Percutaneous Coronary Intervention (PCI) is a high volume procedure with significant morbidity & mortality Percutaneous Coronary Intervention (PCI) is a high volume procedure with significant morbidity & mortality Risk of death in PCI varies widely based on co-morbidities Risk of death in PCI varies widely based on co-morbidities Providing accurate case level estimations can greatly aid patient and physician decision-making Providing accurate case level estimations can greatly aid patient and physician decision-making

4 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 4 Domain Data Quality The American College of Cardiologists has published a standardized data dictionary (ACC-NCDR) and mandates that accredited centers maintain detailed data on all PCI patients The American College of Cardiologists has published a standardized data dictionary (ACC-NCDR) and mandates that accredited centers maintain detailed data on all PCI patients Some states, including Massachusetts, now have mandatory reporting of case data based on the ACC-NCDR Some states, including Massachusetts, now have mandatory reporting of case data based on the ACC-NCDR

5 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 5 Current Risk Model Standard Logistical Regression (LR) Gold standard for risk modeling in interventional cardiology Gold standard for risk modeling in interventional cardiology Type of generalized non-linear model Type of generalized non-linear model –Used in analysis of a binary outcome –Bounded by 0 and 1 Feature (variable) selection Feature (variable) selection –From All Available Data –Known Risk Factors from Prior Studies –Selected Subset of data based on Study Design

6 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 6 Alternative Risk Model Support Vector Machine (SVM) Key Features Key Features –Kernel Functions - introduce non-linearity in the hypothesis space without explicitly requiring a non-linear algorithm LinearLinear PolynomialPolynomial Radial BasedRadial Based –Global Minimum

7 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 7 Risk Model Evaluation Discrimination Provides an estimate of population level accuracy Provides an estimate of population level accuracy Area under the Receiver Operating Characteristic (ROC) Curve Area under the Receiver Operating Characteristic (ROC) Curve Graphed by the sensitivity vs. 1-specificity at different thresholds Graphed by the sensitivity vs. 1-specificity at different thresholds

8 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 8 Risk Model Evaluation Calibration Provides an estimation of case level accuracy Provides an estimation of case level accuracy Hosmer-Lemeshow’s Goodness-of-Fit Test Hosmer-Lemeshow’s Goodness-of-Fit Test –Primarily used in logistic regression –Calculates how well the observed and expected frequencies match –Handles data sparsity better than more common methods (Variance, Pearson’s) –P > 0.05 is a good fit

9 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 9 Source Data Brigham & Women’s Hospital Brigham & Women’s Hospital Interventional Cardiology Database Interventional Cardiology Database January 1, 2002 – October 30, 2004 January 1, 2002 – October 30, 2004 5383 Cases 5383 Cases –Data split two ways each into 2/3 Training (3588) and 1/3 Test (1795) Sequential SplitSequential Split –sorted chronologically –October 27, 2003 split Random SplitRandom Split

10 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 10 Sample Demographics Overview #%Age 0-49 0-4959010.96 50-59 50-59116721.68 60-69 60-69149727.81 70-79 70-79139825.98 80 + 80 +65212.22 Diabetic172131.98 Hypertensive408375.86 Hyperlipidemia373769.44 Prior PCI 182233.85 Salvage Procedure 240.45 Cardiogenic Shock 981.82 Hemodynamic Instability 2654.92 Death781.45

11 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 11 Model Features Age (D) Hyperlipidemia Hx COPD GenderHTN Hx CVD BMI (D) Diabetes Hx PVD Cardiogenic Shock Creatinine (D) Thrombolytic Cardiac arrest Hx CHF IABP Hemodynamic instability CHF EF (D) Smoker Prior MI AMI Prior CABG Prior PCI Procedure urgency (D) Unstable Angina Chronic Angina AMI Within 24 Hours

12 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 12 Logistic Regression Model Development STATA 8.2 (College Station, TX) STATA 8.2 (College Station, TX) Backwards Stepwise Technique Backwards Stepwise Technique Exclusion Threshold (P 0.05 – 0.15) Exclusion Threshold (P 0.05 – 0.15) Feature Selection Feature Selection

13 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 13 Logistic Regression Feature Selection Model development Model development –Sequential Training Set –Stepwise Backwards (P = 0.10) used for feature selection –Stepwise feature removal based on ROC and HL Goodness-of-fit (HL) optimization

14 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 14 Logistic Regression Feature Selection FeatureROC HL P All0.9520.0358 -BMI -BMI0.9520.0706 -EF -EF0.9450.0004 -arrest -arrest0.9510.0602 -hyperlipid -hyperlipid0.94080.0001 -BMI,EF -BMI,EF0.94820.0743 -BMI, Urgency -BMI, Urgency0.9490.1066 -BMI, Urgency, CHF Hx -BMI, Urgency, CHF Hx0.9560.956

15 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 15 Logistic Regression Evaluation TrainingTest ROCHLROCHL 0.150.9460.6720.894<0.001 SEQ0.100.9490.4880.904<0.001 0.050.9360.7040.8890.004 0.150.9260.2690.9200.140 RND0.100.9260.2690.9200.140 0.050.9000.0950.899<0.001

16 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 16 Support Vector Machine Model Development GIST 2.1.1 (Columbia University, NY, NY) GIST 2.1.1 (Columbia University, NY, NY) STATA 8.2 (College Station, TX) STATA 8.2 (College Station, TX) All variables used All variables used Kernel Choice Kernel Choice –Polynomial (1-6) –Radial width factor (related to sigma) (0.1-20) Probabilistic Output Methodology Probabilistic Output Methodology –Discriminant: distance from hyperplane –LR Model using Discriminant as the only feature –Established method to convert SVM classification to regression –Allows use of HL Goodness of fit

17 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 17 SEQ TrainingTest ROCHLROCHL Lin0.9700.5030.8960.003 P20.9910.9660.9070.002 P30.9940.9990.9090.067 P40.9920.9970.9070.163 P50.9870.818 0.8990.713 P60.9760.0490.8850.738 Support Vector Machine Polynomial Evaluation

18 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 18 RND TrainingTest ROCHLROCHL Lin0.9630.6160.8620.817 P20.9920.9200.9000.754 P30.9950.9990.9010.617 P40.9961.0000.9030.521 P50.9960.903 0.8780.749 P60.9970.0130.8710.856 Support Vector Machine Polynomial Evaluation

19 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 19 SEQ TrainingTest ROCHLROCHL R 0.25110.8890.111 R 0.50110.9090.601 R 0.75110.9100.200 R 1.000.9971 0.9100.246 R 1.500.9700.5020.9040.001 R 2.000.9740.8170.9040.001 Support Vector Machine Radial Evaluation

20 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 20 RND TrainingTest ROCHLROCHL R 0.250.99910.8910.046 R 0.50110.9080.593 R 0.75110.9100.199 R 1.000.9971 0.9110.542 R 1.500.9920.961 0.9070.810 R 2.000.8950.9610.8980.232 Support Vector Machine Radial Evaluation

21 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 21 Discussion All Discrimination All Models showed excellent performance All Models showed excellent performance None of the models was significantly different in performance None of the models was significantly different in performance This measure was relatively insensitive to changes in data across widely variable levels of calibration This measure was relatively insensitive to changes in data across widely variable levels of calibration

22 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 22 Discussion LR Calibration For this data, LR was unable to maintain calibration. This is likely due to temporal data drift For this data, LR was unable to maintain calibration. This is likely due to temporal data drift The LR models required manual feature selection and expert knowledge to calibrate the training data sets The LR models required manual feature selection and expert knowledge to calibrate the training data sets

23 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 23 Discussion SVM Calibration Some versions of both kernel types were able to maintain calibration on both data sets Some versions of both kernel types were able to maintain calibration on both data sets Calibration was maintained across larger parameter ranges of both kernels for the random data set than the sequential data set Calibration was maintained across larger parameter ranges of both kernels for the random data set than the sequential data set Current assessments of discrimination and calibration on the training set are insufficient to choose the optimal kernel parameter Current assessments of discrimination and calibration on the training set are insufficient to choose the optimal kernel parameter

24 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 24 Conclusions SVMs could be superior to LR in terms of maintaining calibration over time in this domain SVMs could be superior to LR in terms of maintaining calibration over time in this domain Further exploration is needed to develop additional markers of model robustness Further exploration is needed to develop additional markers of model robustness Further work in evaluating optimal time intervals to create new models or recalibrate old models Further work in evaluating optimal time intervals to create new models or recalibrate old models

25 © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 25 The end

Download ppt "© 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 1 Evaluation of Support Vector Machines for Risk Modeling in Interventional."

Similar presentations

Generative Models Thus far we have essentially considered techniques that perform classification indirectly by modeling the training data, optimizing.

Generative Models Thus far we have essentially considered techniques that perform classification indirectly by modeling the training data, optimizing.
ECG Signal processing (2)

ECG Signal processing (2)
Gall C, Katch A, Rice T, Jeffries HE, Kukuyeva I, and Wetzel RC

Gall C, Katch A, Rice T, Jeffries HE, Kukuyeva I, and Wetzel RC
Guidelines recommend consideration of fibrinolytic therapy if unable to achieve a door to balloon time ≤120 minutes for STEMI patients transferred for.

Guidelines recommend consideration of fibrinolytic therapy if unable to achieve a door to balloon time ≤120 minutes for STEMI patients transferred for.
An Introduction of Support Vector Machine

An Introduction of Support Vector Machine
Support Vector Machines

Support Vector Machines
Particle swarm optimization for parameter determination and feature selection of support vector machines Shih-Wei Lin, Kuo-Ching Ying, Shih-Chieh Chen,

Particle swarm optimization for parameter determination and feature selection of support vector machines Shih-Wei Lin, Kuo-Ching Ying, Shih-Chieh Chen,
The Influence of Radial vs. Femoral Access on Acute Blood Loss in Patients Undergoing Percutaneous Coronary Intervention Amit Nanda 1, Eric Novak MS 2,

The Influence of Radial vs. Femoral Access on Acute Blood Loss in Patients Undergoing Percutaneous Coronary Intervention Amit Nanda 1, Eric Novak MS 2,
Comparison of the New Mayo Clinic Risk Scores and Clinical SYNTAX Score in Predicting Adverse Cardiovascular Outcomes following Percutaneous Coronary Intervention.

Comparison of the New Mayo Clinic Risk Scores and Clinical SYNTAX Score in Predicting Adverse Cardiovascular Outcomes following Percutaneous Coronary Intervention.
Prediction Models in Medicine Clinical Decision Support The Road Ahead Chapter 10.

Prediction Models in Medicine Clinical Decision Support The Road Ahead Chapter 10.
Predictive Automatic Relevance Determination by Expectation Propagation Yuan (Alan) Qi Thomas P. Minka Rosalind W. Picard Zoubin Ghahramani.

Predictive Automatic Relevance Determination by Expectation Propagation Yuan (Alan) Qi Thomas P. Minka Rosalind W. Picard Zoubin Ghahramani.
Using Machine Learning to Model Standard Practice: Retrospective Analysis of Group C-Section Rate via Bagged Decision Trees Rich Caruana Cornell CS Stefan.

Using Machine Learning to Model Standard Practice: Retrospective Analysis of Group C-Section Rate via Bagged Decision Trees Rich Caruana Cornell CS Stefan.
Data mining and statistical learning - lecture 13 Separating hyperplane.

Data mining and statistical learning - lecture 13 Separating hyperplane.
What is Learning All about ?  Get knowledge of by study, experience, or being taught  Become aware by information or from observation  Commit to memory.

What is Learning All about ?  Get knowledge of by study, experience, or being taught  Become aware by information or from observation  Commit to memory.
Lucila Ohno-Machado An introduction to calibration and discrimination methods HST951 Medical Decision Support Harvard Medical School Massachusetts Institute.

Lucila Ohno-Machado An introduction to calibration and discrimination methods HST951 Medical Decision Support Harvard Medical School Massachusetts Institute.
Review Rong Jin. Comparison of Different Classification Models  The goal of all classifiers Predicating class label y for an input x Estimate p(y|x)

Review Rong Jin. Comparison of Different Classification Models  The goal of all classifiers Predicating class label y for an input x Estimate p(y|x)
Validation of predictive regression models Ewout W. Steyerberg, PhD Clinical epidemiologist Frank E. Harrell, PhD Biostatistician.

Validation of predictive regression models Ewout W. Steyerberg, PhD Clinical epidemiologist Frank E. Harrell, PhD Biostatistician.
An Introduction to Support Vector Machines Martin Law.

An Introduction to Support Vector Machines Martin Law.
Ch. Eick: Support Vector Machines: The Main Ideas Reading Material Support Vector Machines: 1.Textbook 2. First 3 columns of Smola/Schönkopf article on.

Ch. Eick: Support Vector Machines: The Main Ideas Reading Material Support Vector Machines: 1.Textbook 2. First 3 columns of Smola/Schönkopf article on.
Performance Reports Andy Bindman MD Department of Medicine, Epidemiology and Biostatistics UCSF.

Performance Reports Andy Bindman MD Department of Medicine, Epidemiology and Biostatistics UCSF.

Similar presentations

Ads by Google