1. Comparing correlated correlations Advisor: Rhonda Decook Client: Vinayak Consultants: Tianyu Li, Qinbin Fan Department of Statistics and Actuarial Science University of Iowa

2. LOGO Outline  Introduction  Data Highlights  Results & Analysis  Conclusion

3. LOGO Introduction  Gold Standard: first take the average score of each image from 3 graders and then re-rank them. ( we also tried other ways to define the gold standard, but this definition is the one we mainly use)  Let r alg.i,GS represent the pearson correlation coefficient between algorithm i and the gold standard (GS).  The r alg.i,GS values for the data set with 12 algorithms and 25 images are shown below (from largest correlation to smallest):

4. LOGO Introduction Correlation with AlgorithmGS 12*0.6583 100.6259 30.5806 90.5718 40.5385 80.5040 60.5000 50.4630 70.4601 10.4300 110.3104 20.2422

5. LOGO Introduction

6. LOGO  We use the bootstrap to form the confidence interval on the difference between the transformed version of r.  The bootstrap method takes into account the fact that the algorithms were all applied to the same set of 25 images.  The bootstrap method resamples with replacement from the original set of n=25 images, to create a ‘new hypothetical’ data set. We calculate the difference in correlations in each of 5000 bootstrapped data sets to provide us with sampling distribution for the difference Introduction

7. LOGO Introduction

8. LOGO Data Highlights Patient ID12… A1.161542.114705… B1.2031862.126865… ………… 123GS 16211316 20132320 …………

9. LOGO Results & Analysis r12-rj (raw) z12-zj (fisher) CI of Diff (99.5%) CI of Diff (95%) Significan t Diff A2 VS A12 0.4160.542( 0.042, 1.338 )( 0.169, 1.048 )YES A11 VS A12 0.3470.468(-0.109, 1.266 )( 0.087, 0.975 )NO/YES A1 VS A12 0.2280.329(-0.253, 0.977 )(-0.058, 0.761 )NO A7 VS A12 0.1980.292(-0.229, 0.870 )(-0.046, 0.664 )NO A5 VS A12 0.1950.288(-0.255, 0.845 )(-0.066, 0.670 )NO A6 VS A12 0.1580.240(-0.466, 0.835 )(-0.208, 0.620 )NO

10. LOGO Results & Analysis A8 VS A12 0.1540.235(-0.310, 0.753 ) (-0.113, 0.596 ) NO A4 VS A12 0.1190.187(-0.431, 0.768 )(-0.211, 0.584 )NO A9 VS A12 0.0860.139(-0.461, 0.689 )(-0.256, 0.504 )NO A3 VS A12 0.0770.126(-0.422, 0.652 )(-0.222, 0.464 )NO A10 VS A12 0.0320.055(-0.411, 0.576 )(-0.255, 0.392 )NO

11. LOGO Results & Analysis  A2(worst) VS A12

12. LOGO Results & Analysis A11(2 nd worst) VS A12

13. LOGO Results & Analysis  We also tried other ways to define the gold standard, for example, we tried to use only the 2 most strongly correlated columns and do the same, to use the median rank of all 3 for each image, and do the same, and to use averages without re-ranking. We found pretty similar results from all four cases. To save space and time, we do not present results of the other three.

14. LOGO Conclusion  We did not find statistically significant difference between our client’s method and all the other methods.  Results may vary when we have more data. ( more image grades)