Developing a Validated Tool For Evaluating Whole Slide Images

University of Pittsburgh Department of Biomedical Informatics Authors Dana Grzybicki, MD, PhD Evaluation Team – Principal Investigator Russell Silowash, BS Evaluation Team – Research Analyst Robb Wilson, MA Evaluation Team – Project Manager Leslie Anthony, MA UPMC IMITS Telepathology Project – Project Manager

University of Pittsburgh Department of Biomedical Informatics Background Through appropriations in the defense-spending bills for 2002 and 2004, the University of Pittsburgh Medical Center (UPMC) and the United State Air Force Medical Service (AFMS) created a partnership called the Integrated Medical Information Technology System (IMITS) Program Telepathology is a branch of the IMITS program that implements and validates digital pathology practices

University of Pittsburgh Department of Biomedical Informatics Introduction Purpose of evaluation research –Independent examination of questions related to: Technical validity Feasibility Effectiveness To our knowledge, there are no generally available validated tools for evaluating WSI cases The UPMC Digital Pathology Imaging Group is working on the validation of a unique evaluation tool

University of Pittsburgh Department of Biomedical Informatics Participants The validation of this tool was part of our telepathology evaluation project involving 5 UPMC pathologists –2 pathology fellows –3 staff pathologists with training in GU pathology

University of Pittsburgh Department of Biomedical Informatics Case Selection 30 difficult prostate biopsy foci DiagnosisNumber of Cases Adenocarcinoma12 Atypical6 Atypical PIN (ATYPIN)1 High Grade PIN3 Benign8

University of Pittsburgh Department of Biomedical Informatics

University of Pittsburgh Department of Biomedical Informatics

University of Pittsburgh Department of Biomedical Informatics Hypotheses Content Validity –The assessment is asking the proper questions for the study at hand Internal Validity –There will be a positive correlation between the number of slides/images in a case and the time needed to complete the case

University of Pittsburgh Department of Biomedical Informatics Hypotheses (continued) External Validity –Whole slide image quality will be positively correlated with glass slide quality Construct Validity –There will be a negative correlation between the diagnostic confidence of a participant and the case complexity rating

University of Pittsburgh Department of Biomedical Informatics Content Validity Results Content validity has been obtained by gaining feedback from pathologists that are part of the Digital Pathology Imaging Group (DPIG)

University of Pittsburgh Department of Biomedical Informatics Internal Validity Results Statistically significant for only one participant (r 2 =0.327, p<0.01) Our inability to demonstrate internal validity for most of the participants was most likely due to time categorical variables that were too broad

University of Pittsburgh Department of Biomedical Informatics External Validity Results A positive correlation exists between WSI quality and glass slide quality There were statistically significant positive correlations for 3 of the 5 subjects

University of Pittsburgh Department of Biomedical Informatics Construct Validity Results Negative correlation exists in WSI phase of study between confidence in diagnosis and case complexity 3 of 5 pathologists had statistically significant positive correlations in the WSI Phase

University of Pittsburgh Department of Biomedical Informatics Construct Validity Results (continued) Negative correlation exists in the Glass phase between case complexity and diagnostic confidence 3 of 5 pathologists had statistically significant positive correlations in the Glass Phase

University of Pittsburgh Department of Biomedical Informatics Summary We are able to establish content, external and construct validity, however internal validity at this point has not been definitively established. –Low internal validity could be due to time categories being too broad

University of Pittsburgh Department of Biomedical Informatics Next Steps Change the categories for the time variable Time to Complete (Minus Interruptions) Less Than 15 Minutes 15 – 30 Minutes 30 – 45 Minutes 45 – 60 Minutes Over 60 Minutes Time to Complete (Minus Interruptions) Less Than 30 Seconds7 – 10 Minutes 30 Seconds – 1 Minute10 – 12 Minutes 1 – 2 Minutes12 – 15 Minutes 2 – 5 MinutesOver 15 Minutes 5 – 7 Minutes Implement an automatic timing solution

University of Pittsburgh Department of Biomedical Informatics Next Steps (continued) Continue the validation of this tool by obtaining additional data and testing internal validity utilizing the modified time variables

University of Pittsburgh Department of Biomedical Informatics Funding This work was supported by funding from the U.S. Air Force administered by the U.S. Army Medical Research Acquisition Activity, Fort Detrick, Maryland (Award No. W81XWH and Contract No. DAMD ). The content of the information does not imply U.S. Air Force or Government endorsement of factual accuracy or opinion