1. Effects of structured template reporting on the quality of radiology reports Juan Carlos Batlle, MD Giles W Boland, MD Daniel I Rosenthal, MD Partners Research Retreat March 24, 2009 Originally presented at RSNA December 2, 2008

2. Background Structured reporting increasingly used as a means of systematizing reports and making them clinician and “computer” friendly Stillman 2008 JACR 51; Taira 2001 Radiographics 21 Transition from free dictation to structured reporting will involve structured templates for free dictation Effects of templates on reporting are uncertain

3. Purpose To evaluate surrogate objective metrics for assessment of text quality of radiology reports To assess the change in text quality of reports after a department-wide transition to automatic structured templates issued by the department

4. Brief Methods 1 In early 2007, MGH Radiology transitioned to Commissure platform (Nuance Communications) voice-recognition dictation software Previous dictation software allowed personal templates (“macros”) used in addition to free-dictation Commissure allows automatic “population” of a structured template into the dictation upon opening the patient exam, according to exam code (e.g. CTABP+, CTCH+, etc.). Structured templates for each exam code created by division committee

5. Brief Methods 2 OLAP (Online Analytical Processing) Cube used to extract dictations of all abdominopelvic CTs interpreted by three randomly-selected radiologists from 2006 and 2007 (n=9594) into Excel. Comparison was made of the text of the reports before (n=5100) and after (n=4494) a transition to automatically populated structured templates. Text was analyzed for word count, sentence structure, and Flesch Reading Ease by semi- automated computer software. Flesch Reading Ease rates text on a 100-point scale based on the average number of syllables per word and words per sentence.

6. Results After the transition to structured reports: Significant decrease in word count (292 to 226 words, p<0.001) Significant increase in number of words per sentence (4.8 to 5.0, p<0.001) Significant worsening in Flesch reading ease score (30.6 to 24.6, p<0.001)

7. Word Count Significantly Decreased After Transition

8. Readability Significantly Decreased After Transition

9. Follow-up Data Preliminary assessment of two randomly selected radiologists from the thoracic imaging division: Attending 1 Word count decrease from 266 to 231 Words/sentence increase from 4.9 to 5.4 Attending 2 Word count decrease from 246 to 232 Words/sentence increase from 5.2 to 5.5

10. Limitations Small group of radiologists studied, though large sample size Metrics of text “quality” are imperfect surrogates for true quality measurement

11. Conclusions Implementation of structured templates led to significant changes as measured in the text of dictations. This may significantly influence overall readability of reports and may be unanticipated Future attention should be paid to optimizing structured templates according to surrogate metrics of text quality Ongoing assessment of these metrics may be performed easily using OLAP cube software