Copyright restrictions may apply JAMA Ophthalmology Journal Club Slides: Centralized Grading of Retinopathy of Prematurity Daniel E, Quinn GE, Hildebrand PL, et al; e-ROP Cooperative Group. Validated system for centralized grading of retinopathy of prematurity: Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study. JAMA Ophthalmol. Published online March 26, doi: /jamaophthalmol

Copyright restrictions may apply Good neonatal care, especially in emerging world economies with increasing survival of preterm low-birth-weight infants, is propagating an epidemic of retinopathy of prematurity (ROP), elevating ROP into a leading cause of preventable childhood blindness. Ophthalmologists skilled in detection of ROP are scarce. Remote screening methods are feasible options for focusing this limited expertise on diagnostic examinations of infants at risk and needing treatment. Nonphysician trained readers (TRs) of ROP images can substantially reduce cumulative diagnostic screening time of ROP experts if their grading is valid and reliable. Objective –To describe a centralized system for grading digital images of ROP by nonphysician TRs in the Telemedicine Approaches to Evaluating Acute- Phase Retinopathy of Prematurity (e-ROP) Study. Introduction

Copyright restrictions may apply Study Design –Multicenter observational cohort study. Participants –Infants with birth weight <1251 g. Data Analysis –Intergrader and intragrader agreement calculated using exact percentage of agreement and weighted κ. –Weighted κ was calculated by a weighting matrix specific to each grading item in which discrepant grades were assigned partial credit for agreement depending on how close they were, and “ungradable” was given 25% agreement with all other grades. –95% CI for the weighted κ was calculated using the bootstrap. Methods

Copyright restrictions may apply TRs from diverse undergraduate backgrounds went through 3-phase training. –Phase 1: Didactic lectures, interactive sessions, and reading assignments; visit to neonatal intensive care unit at The Children’s Hospital of Philadelphia to observe imaging of premature babies; and knowledge assessment testing. –Phase 2: TRs independently viewed and graded training image sets with known ROP grading from another study; used a paper grading form; and had training sessions graded, reviewed, and discussed to determine areas that warranted additional training. –Phase 3: TRs graded and reviewed 100 ROP image sets using electronic form and grading protocol and met with study chair, reading center (RC) director, and a clinical expert. The e-ROP grading protocol detailed criteria for evaluation of image quality and the key morphologic features of ROP. A group of 4 experts (retinal specialists experienced in ROP) and the RC director generated consensus final grading results for use in assessing TR performance. Certified when agreement with consensus final grading was >80%. Methods

Copyright restrictions may apply Methods: Study Flow ENROLLMENT 1285 Infants enrolled DIAGNOSTIC EXAM 1257 Infants had 4263 diagnostic examinations 244 Infants had RW-ROP 1013 Infants did not have RW-ROP IMAGING 242 Infants with RW-ROP imaged All 242 infants with RW- ROP image sets selected 999 Infants without RW-ROP imaged 613 Infants without RW-ROP had image sets selected GRADING 1759 Image sets from 454 eyes with RW-ROP selected 150 Image sets from 30 eyes without RW- ROP selected 3611 Image sets from 1226 eyes without RW-ROP selected 5250 Image sets from 855 eyes of 1710 infants graded RW-ROP indicates referral-warranted ROP.

Copyright restrictions may apply Grading of Study Images –Standard 6-image sets acquired for each eye were graded independently by 2 TRs, with discrepancies adjudicated by RC director. –Grading performed at standardized independent workstations with secure Internet access and similarly configured computers with monitors calibrated every 2 weeks to maintain consistency in brightness and hue. –Software developed for displaying and manipulating contrast, brightness, and magnification in the ROP images. –Data from grading captured using web-based forms. –TRs masked to all infant demographic information including birth weight and gestational age, clinical data on ROP findings from the diagnostic eye examination, and the grading results from image sets of previous visits and image sets from the fellow eye. Methods

Copyright restrictions may apply Methods Screenshot of e-ROP Image Display and Web-Based Forms for Grading

Copyright restrictions may apply Adjudication for Components of RW-ROP ComponentImages, No. Images With Any Adjudication for Component, No. (%) Total RC grading (56.4) From RC final grading RW-ROP No (12.9) Yes (52.6) Cannot determine11460 (52.6) Total (24.5) Plus disease No (2.3) Yes13071 (54.6) Cannot determine7520 (26.7) Total (3.9) Zone I ROP No (8.4) Yes (56.7) Cannot determine10036 (36.0) Total (12.4) Stage 3 or worse ROP No (11.5) Yes (31.3) Cannot determine9438 (40.4) Total (16.9) Results

Copyright restrictions may apply TR Grading Variability of Morphology Variable Intergrader Variability a Intragrader Variability b Agreement, %Weighted ĸ (95% CI)Agreement, %Weighted ĸ (95% CI) Abnormal posterior pole vessels ( ) ( ) Total quadrants of plus or preplus disease ( ) ( ) Total quadrants of plus ( ) ( ) Dominant feature ( ) ( ) Any ROP ( ) ( ) Demarcation line ( ) ( ) Ridge ( ) ( ) Extraretinal fibrovascular proliferation ( ) ( ) Flat preretinal neovascular proliferation ( ) ( ) Retinal detachment ( ) ( ) Highest stage ( ) ( ) Lowest zone ( ) ( ) Plus disease ( ) ( ) Zone I ROP ( ) ( ) Stage 3 or worse ROP ( ) ( ) RW-ROP ( ) ( ) a Combined contemporaneous variable sample graded every 3 months (n = 80). b Repeated grading of 25 image sets for temporal drift. Results

Copyright restrictions may apply TR Grading Variability of Image Quality Image Quality Intergrader Variability a Intragrader Variability b Agreement, %Weighted κ (95% CI)Agreement, %Weighted κ (95% CI) Pupil ( ) ( ) Disc center ( ) ( ) Disc up ( ) ( ) Disc down ( ) ( ) Disc temporal ( ) ( ) Disc nasal ( ) ( ) Results a Combined contemporaneous variable sample graded every 3 months (n = 80). b Repeated grading of 25 image sets for temporal drift.

Copyright restrictions may apply A system to evaluate the competency of remote nonphysician TRs had not been detailed in prior ROP telemedicine studies. e-ROP RC: –Developed an ROP curriculum for training and certification for nonphysician readers. –Developed and implemented a standardized grading protocol using electronic data capture. –Established a standard criterion of reference for RW-ROP morphology, any ROP, and preplus disease in retinal digital images through a process of integrating the grading of 3 expert readers, the RC director, and the study chair and using this for comparing TR grading during certification. The excellent agreement between TRs reflects the extensive and rigorous training and certification process. Comment

Copyright restrictions may apply This study’s data suggest that the e-ROP system for training and certifying nonphysicians to grade ROP images under the supervision of an RC director reliably detects potentially serious ROP with good intragrader and intergrader consistency and minimal temporal drift. Zone I ROP –Intragrader agreement least in identifying zone I ROP attributable largely to difficulty in accurately identifying the foveal center in the images. –Consistent with the results from a study that reported large variability in identifying the foveal center by ROP-specialized ophthalmologists. –The reliability of identifying the foveal center and subsequent delineation of zone I consistently in digital images could be increased by using a standard zone I template for digital images. Enhancing the appearance of the ROP morphology and attenuating background noise in poor-quality images by manipulating the contrast, brightness, magnification, and gray tone appear to bring more certainty to detecting ROP pathology in digital images; this will be tested in a future study. Comment

Copyright restrictions may apply Plus Disease –Identifying plus disease had an intergrader variability weighted κ of –International Classification of Retinopathy of Prematurity images as standards for tortuosity and dilation in identifying plus and preplus disease do not appear to adequately minimize intergrader variability. –Identification of plus disease among ROP experts appears to be highly variable over several previous studies. –These disagreements in identifying plus disease that persist in telemedicine ROP studies need more rigorous refinements on the definition and quantitative methods of detecting plus disease in digital images. Comment

Copyright restrictions may apply Limitations of the Study –Readers had no access to information on the gestational age, birth weight, or findings in the fellow eye, which could have improved the sensitivity and specificity in the study. –No gold standard to assess the competency of the TR in identifying morphological features in the retinal images; the consensus opinion of a few experts in ROP, subject to error, was used as the standard for comparison for training and certifying TRs. To our knowledge, this is the first study that has demonstrated consistent and good agreement between and among nonphysician TRs grading ROP from digital images using a centralized reading facility. Comment

Copyright restrictions may apply If you have questions, please contact the corresponding author: –Ebenezer Daniel, MBBS, MS, MPH, PhD, Ophthalmology Reading Center, Department of Ophthalmology, University of Pennsylvania, 3535 Market St, Ste 700, Philadelphia, PA Funding/Support This work was supported by cooperative agreement grant U10 EY from the National Eye Institute. Conflict of Interest Disclosures All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Hildebrand reported receiving consulting fees from Inoveon Corp, serving as chairman of the board of directors for Inoveon Corp, and receiving royalties for US patents and , “Digital Disease Management System” (assignee: Board of Regents, University of Oklahoma). Dr Ells reported serving as a member of the scientific advisory board for Clarity Systems. Dr Hubbard reported receiving payment from the University of Pennsylvania as an expert grader of photographs in this work and receiving consulting fees from VisionQuest Biomedical, LLC for grading photographs outside this work. Dr Capone reported being a founding partner of FocusROP, LLC. Dr Ying reported serving as a statistical consultant for Janssen Research and Development, LLC. No other disclosures were reported. Contact Information