1. Foundation.cap.org SlideTutor Project – Using Virtual Slides to Understand How Pathologists Think and Learn Rebecca Crowley, MD, MS Saturday, April 16, 2011

2. Microscopic diagnosis is one of the most complex visual classification tasks that humans perform In domains outside of medicine, intelligent computer-based training is common − Aviation, nuclear and power industries, and the military Background - I © 2011 College of American Pathologists. All rights reserved. 2

3. Research has shown that these systems can approach benefits of one-on-one teaching Advances in virtual slide and educational technologies may help us to rethink the platform we use to educate pathologists Background - II © 2011 College of American Pathologists. All rights reserved. 3

4. Build the most advanced medical training system possible with current technology Use the system to more efficiently train pathology residents and fellows Understand how we learn the complex task of microscopic diagnosis Predict future performance (including errors Goals of the SlideTutor Project © 2011 College of American Pathologists. All rights reserved. 4

5. Q1. How do pathologists develop expertise in microscopic diagnosis?

6. Early studies of expertise © 2011 College of American Pathologists. All rights reserved. 6

7. 7 Overall accuracy Crowley et al. JAMIA, 2003 © 2011 College of American Pathologists. All rights reserved.

8. Resident looking at DCIS © 2011 College of American Pathologists. All rights reserved. 8 Intermediate I5: 20 and some of the ducts that are expanded with small cells 21with focal, possibly central, area of necrosis. 22So just scan this slide around and try to determine some focal areas that I want to concentrate and focus on. 23Now I’m looking at some of the ducts that are expanded. 25 And some of these ducts, they also have holes, 26 and these are sort of punched-out holes, 27 very uniform, which… 29 So at this magnification, I think it is a DCIS. identify-histologic-cue statement-of-hypothesis

9. Attending looking at DCIS © 2011 College of American Pathologists. All rights reserved. 9 Expert E7: Okay let’s look at low power I think the tissue is breast I recognize some normal Here is in situ carcinoma I have to find out if there is any invasion identify-histologic-cue identify-anatomic-location statement-of-hypothesis set-goal-identify-feature

10. 10 Process Differences Crowley et al. JAMIA, 2003 © 2011 College of American Pathologists. All rights reserved.

11. Task Errors Crowley et al. JAMIA, 2003 11 © 2011 College of American Pathologists. All rights reserved.

12. 12 Task Analysis Latencies Crowley et al. JAMIA, 2003 © 2011 College of American Pathologists. All rights reserved.

13. Q2. What makes our system different than other virtual-slide educational systems?

14. Student Model Pedagogic Knowledge Interface Expert Module Allow correct steps Correct errors Give hints on next step Collect data on what student does Make predictions on what student knows Provide data for pedagogic decision making Case sequence When to intervene How much to intervene How to intervene 14/47 © 2011 College of American Pathologists. All rights reserved.

15. Student draws another link. Object is red and bug message explains error Student draws between evidence and hypothesis Selects hypothesis Student specifies feature further to make it more meaningful evidence Tutor permits correct answer 3. Selects finding 1. Student indicates new finding 2. Clicks on finding, creating ‘x’ System moves viewer or moves and annotates the area for students Hints are hierarchically structured providing increasing help Student has already identified some evidence

16. Diagnostic Reasoning

17. Reporting with CAP Protocols © 2011 College of American Pathologists. All rights reserved. 17

18. Calibrating Confidence © 2011 College of American Pathologists. All rights reserved. 18

19. © 2011 College of American Pathologists. All rights reserved. 19 Movie of SlideTutor

20. Q3. How effective is the system? © 2011 College of American Pathologists. All rights reserved.

21. Errors decline Crowley et al. JAMIA, 2007 © 2011 College of American Pathologists. All rights reserved.

22. 22/47 Test performance improves *** Crowley et al. JAMIA, 2007 © 2011 College of American Pathologists. All rights reserved.

23. 23/47 Diagnostic accuracy improves *** Crowley et al. JAMIA, 2007 © 2011 College of American Pathologists. All rights reserved.

24. Measures of calibration 24/47 Saadawi et al, Adv in Health Sci Ed, 2010 © 2011 College of American Pathologists. All rights reserved.

25. 25/47 Calibration improves Test 1Test 2Test 1 to 2 DeltaFp- value G Correlation 0.570.740.175.040.04 Bias 0.13 0.00.070.79 Discrimination 0.300.420.125.920.02 Saadawi et al, Adv in Health Sci Ed, 2010

26. 26/47 Community Pathologists Crowley et al Proceedings of ITS 2010

27. 27/47 Adherence to CAP improves Crowley et al Proceedings of ITS 2010

28. Q4. Can we predict subsequent diagnostic performance and competency? © 2011 College of American Pathologists. All rights reserved.

29. Modeling and Prediction Store every action and it’s meaning Create learning curves for skills and knowledge Predict whether residents will be able to identify a particular finding or make a particular diagnosis based on previous performance © 2011 College of American Pathologists. All rights reserved.

30. 30/47 Bayesian Models of Skill Acquisition Yudelson, Medvedeva, and Crowley. User Modeling and User Adapted Interaction, 2008 © 2011 College of American Pathologists. All rights reserved.

31. Modeling and Prediction © 2011 College of American Pathologists. All rights reserved. 31

32. Accuracy of our models Models that we have developed are reasonably accurate Current work is focused on predicting diagnostic accuracy using both visual and symbolic information - Where and how the resident ‘looks’ at a slide can help us predict the kinds of mistakes they may make 32/47 © 2011 College of American Pathologists. All rights reserved.

33. Conclusions Pathologists learn the extraordinarily complex task of microscopic diagnosis in a rather predictable fashion that can be studied We can use this information to develop highly advanced medical training systems in our domain 33/47

34. Conclusions These systems could significantly enhancement to training future pathologists Information we can obtain from the system can be used to make much more detailed assessments of competency than are currently available 34/47

35. Future landscape Electronic books tied directly into systems that provide practice Improved competency based training and assessment Diagnostic decision support that builds on ‘awareness’ of the pathologist as a unique individual 35/47

36. http://slidetutor.upmc.edu © 2011 College of American Pathologists. All rights reserved.

37. 37 Acknowledgements o Collaborators: − Drazen Jukic, MD, PhD, Alka Palakar, MD − Gilan Saadawi, MD, PhD, Claudia Mello-Thoms, PhD − Roger Azevedo, PhD (McGill University) − George Xu, MD, PhD and Michael Feldman MD, PhD (University of Pennsylvania) − Dana Grzybicki, MD, PhD, Steve Raab, MD (University of Colorado) o Programmers: − Olga Medvedeva − Eugene Tseytlin − Girish Chavan o Research Associates: − Elizabeth Legowski, Kayse Gearhart o Knowledge Engineer − Melissa Castine o Students: − Velma Payne o Communications: − Karma Edwards o Funding: − National Library of Medicine 1R01 LM007891 and 2R01 LM007891 (ARRA) − National Cancer Institute 1R25 CA101959 and 2R25 CA101959 − Agency for Health Care Research and Quality I UL1 RR024153-01(Grzybicki, PI) − National Library of Medicine Training Grant 5-T15-LM07059 − University of Pittsburgh Competitive Medical Research Fund © 2011 College of American Pathologists. All rights reserved.