1 Diagnosing Breast Cancer with Ensemble Strategies for a Medical Diagnostic Decision Support System David West East Carolina University Paul Mangiameli University of Rhode Island Rohit Rampal Portland State University

2 Introduction Breast cancer is one of the most prevalent illnesses to women over 40 Society must do all it can to reduce the frequency and severity of this disease Early and accurate detection is critical Our MDSS aids physicians in this task

3 Introduction Critical to performance and acceptance of any MDSS is the selection of the underlying models This paper investigates the performance of MDSS comprised of: –single (individual) models and –ensembles (groups) of models How should ensembles be formed

4 Overview of Presentation Discussion of Literature on Model Selections for MDSS Research Methodology –Breast cancer data sets –Model description Experimental Design Results of Generalized Classification Errors Greedy Model Selection Strategy Conclusion

5 Literature Review Most MDSS studies use single best model strategy –Parametric: Linear discriminant analysis Logistic regression –Non-parametric K nearest neighbor Kernel density

6 Literature Review –Neural Networks Multilayer perceptron Radial Basis Function Mixture of experts –Classification and Regression Trees (CART)

7 Literature Review A very limited number of studies use ensembles of models The ensembles are comprised of one type of model – we call these baseline ensembles These ensembles are usually bagging or bootstrap aggregating ensembles –Bootstrap is a way of perturbing the data so as to create diversity in the decision making of each model in the ensemble

8 Research Methodology Breast Cancer Data Sets Model Descriptions Experimental Design

9 Breast Cancer Data Sets Cytology Data –699 Records of fine needle aspirants –Biopsied tumor mass is benign or malignant 458 benign and 241 malignant Prognostic Data –198 Records of invasive, non-metastasized, malignant tumors –Tumor is either recurrent or non-recurrent 151 non-recurrent and 47 recurrent

10 Model Description 24 Models in Total Linear Discriminant Analysis (LDA) - 1 Logistic Regression (LR) - 1 Nearest Neighbor Classifiers (KNN) - 4 Kernal Density (KNN) - 4 Multilayer Perceptron (MLP) - 4 Mixture of Experts (MOE) - 4 Radial Basis Function (RBF) - 4 Classification and Regression Trees (CART) - 2

11 Experimental Design Single Best Model Generalization Error Split data into 3 partitions –Training set –Validation set –Independent test set - test set is randomly selected and equals 10% of data Partition remaining data into 10 mutually exclusive sets – ten fold cross validation One partition is the validation set

12 Experimental Design Single Best Model Generalization Error Collapse other 9 partitions and use as training set Train each of the 24 models and then compute error on the validation set Repeat 10 times using each partition as the validation set

13 Experimental Design Single Best Model Generalization Error Model with the lowest error across the 10 fold cross validation runs is called the single best model Test the single best model against the hold out test set and compute the error Repeat previous steps 100 times Determine Single Best Model generalized error over all 100 runs

14 Experimental Design Baseline Bagging Ensembles Bagging means Bootstrap Aggregating Baseline means comprise of just one of the 24 models There are 24 baseline bagging ensembles – one for each model

15 Experimental Design Baseline Bagging Ensembles Essentially, the same ten fold cross validation approach is used Perturb the training data (sample with replacement) for each of the 24 models in the ensemble –Creates 24 uniquely weighted models with the same general architecture within the ensemble Use majority vote to determine aggregate decisions for the test set Use 500 runs to determine mean generalization error for each ensemble

16 Experimental Design Diverse Bagging Ensembles Same as the baseline bagging procedure but now pre-select the models in the ensemble Of the 24 models, randomly choose: 24, 12, 8, 6, 4, 3, and 2 different models to make up the ensemble. Also, restrict the choice of models for the ensemble to the top 50% and top 25% of the single best models

17 Results Generalized Error Initial Results Design and Results of Greedy Algorithm

18 Initial Results Comparison of Generalized Error Model Selection Strategy Prognostic DataCytology Data Single Best Model Baseline Bagging Ensemble Diverse Random Ensembles Top 50% Diverse Bagging Ensemble Top 25% Diverse Bagging Ensemble

19 Discussion of Initial Results The existing literature uses either the single best or baseline bagging model selection strategy By using a diverse initial group of 24 models and then creating diverse ensembles comprised of top 25% of the single best models, the generalized errors were significantly lowered

20 Some Observations about Initial Results Large number of single models were the best at least once for the 100 runs Baseline bagging ensembles did poorly for the cytology data compared to the single best model –We found near unanimous consensus (indicated by high levels of error correlations) among models –Indicates that bootstrapping may not create model instability The best diverse ensembles were comprised of 3 different models and the worse had all had more than 6

21 Development of Greedy Ensemble Algorithm Create ensembles with: Diversity of models (3 - 6 different models in the ensemble) Each model has a low generalization error in the single best model results Each model had low error correlation levels (i.e.- high model instability) in the baseline bagging results

22 Model Selection for Greedy Algorithm Regressing generalized error as a function of model instability we found that there were a few models with high instability and low generalized error for each data set We choose 3 different models for the greedy algorithm ensemble Prognostic - RBFd, LR, CARTb Cytology – RBFc, RBFb, MOEa

23 Results Comparison of Greedy Algorithm Ensemble to Top 25% Ensemble Model Selection Strategy Prognostic DataCytology Data Top 25% Diverse Bagging Ensemble Greedy Algorithm Ensemble

24 Concluding Discussion

25 Investigated the Performance of MDSS Comprised of: Single models Baseline bagging ensembles Diverse random bagging ensembles Selected bagging ensembles Greedy Algorithm bagging ensembles

26 We Found that MDSS comprised of a single model or an ensemble of single models (baseline) do not perform well Diverse ensembles are better Diverse ensembles comprised of models with low generalized errors from the single best model results are still better The best results are achieved by diverse ensembles comprised of models with low generalized error from the single best model results and high instability from the baseline bagging results

27 Limitations Limitations of the Methodology –Use of plurality voting –Use of only bootstrap Limitations of Applicability –Only 2 data sets Both data sets regard breast cancer –MDSS is crude – not for commercial applications