1. Clinical Area of Focus: Radiology Group Members: Dr. Nadja Kadom, Mohammadali Mojarrad, Kristin McDougall Faculty Advisor: Dr. Nadja Kadom By Mohammadali Mojarrad

2. SYSTEM ERRORS  Organizational  Clustering  Faulty medical history  Teamwork/communication  Inefficient processes  Management/supervision  Policy/procedures  Technical errors  Unavoidable errors COGNITIVE ERRORS  Faulty information processing  Faulty interpretation  Premature closure  Over-interpretation  Faulty context generation  Failure to order f/u  Faulty data gathering  Faulty test technique  Ineffective history review/exam  Faulty knowledge base  Inadequate skills  Perceptual error  Under-interpretation 2011 G. Taylor

3. SYSTEM ERRORS  Organizational  Clustering  Faulty medical history  Teamwork/communication  Inefficient processes  Management/supervision  Policy/procedures  Technical errors  Unavoidable errors COGNITIVE ERRORS  Faulty information processing  Faulty interpretation  Premature closure  Over-interpretation  Faulty context generation  Failure to oredr f/u  Faulty data gathering  Faulty test technique  Ineffective history review/exam  Faulty knowledge base  Inadequate skills  Perceptual error  Under-interpretation 2011 G. Taylor

4. Hawkins 2014

5.  Radiology reports are an important means of communication between radiologists and other health care providers.  Voice recognition (VR) software has largely replaced transcriptionists and decreased report turnaround times.  Voice recognition software can cause errors in radiology reports that can …  Harm patients  Cause a negative perception of the radiologist  Delay report finalization, cause clinician calls

6. Error rates using voice recognition (VR) versus conventional transcriptions Pezzullo et al. 2008  VR reports take 50% longer to complete despite being 24% shorter compared to transcribed reports  5.1 errors /case  90% contain errors before sign-off (vs 10% transcribed)  35% errors AFTER sign-off

7. Error rates: 35% of reports with errors (Pezzulo et.al. 2008) 36 % of reports with errors (Chian et al. 2010) 22% of reports with errors (Quint* et al. 2008) *Radiologist error rates from 0-100% *No difference native vs. non-native *No difference faculty alone vs faculty/trainee

10.  Radiologist  Accuracy at stake  Reputation at stake  Kristin Girard-McDougall (GE)  RIS manager  Software expert  Mohammadali Mojarrad  Willing to put in hours

11. What are we trying to accomplish?  What?  For whom?  How good?  By when? I am going to improve my reports.

12.  We will reduce the number of voice recognition errors in Dr. Kadom’s reports by 20% by August 31, 2014.  Possible applicability to other radiologists though errors, dictation styles, and proofreading habits differ  Plan to share results locally/nationally to inspire other radiologists to do QI on their VR errors

13. Decrease VR errors by 20% Radiologist’s performance Technology performance

14.  2013 ARRS lecture video  David L. Weiss, Imaging Informatics, Carilion Clinic, Roanoke, VA  Dictation style  Navigation  Microphone  Macros & templates  Vendor selection Plosives, know your software Keep eyes on study Switch off, noise masking, room design Helpful, use as many as possible Important

15. Decrease VR errors by 20% Radiologist’s performance Technology performance Quality of mic Quality of software Background noise Trainable Quality of mic Quality of software Background noise Trainable Proof reading Enunciation Use of macros Switch off mic Proof reading Enunciation Use of macros Switch off mic

16. Decrease VR errors by 20% Radiologist’s performance Technology performance Quality of mic Quality of software Background noise Trainable Quality of mic Quality of software Background noise Trainable Proof reading Enunciation Use of macros Switch off mic Proof reading Enunciation Use of macros Switch off mic

17. VR error rate System PeopleTechnology Background noise interference Ineffective Voice recognition Accent Limited Time Lack of adequate proof reading Work space noise Work load Fatigue No resident Lack of transcriptionist or Proof reader Cost Microphone

18. IdeasSelectedReasoning Proof readingXNecessary ChecklistXHelp build new proof reading habit Fix “disk” errorXEasy fix in software Fix “insert macro” errorXInsert manually rather than dictate User profile resetXGE request Conscious dictationXNecessary Background noiseCannot improve Notify GE of issuesVoice files deleted, GE cannot review New softwareNo money

19.  Outcome measure  % of reports with errors  Process measure  Proofreading evidenced by observer  Balancing measure  Time spent proofreading

20. Monday (n=428) Saturday (n=518) Total (n=946) Number of reports with errors 157 (37%)269 (52%)426 ( 45%) July 2013 – June 2014 1 Monday and 1 Sunday each month= 12 Mondays & 12 Sundays

21. Baseline Goal

22. May 1: Medicolegal ARRS May 22: GE reset July: Checklist Goal

23.  History: spelling errors  Technique: contrast dose  Findings: Proofread  No displaced  Nondisplaced  Conus  colon is  Colon  :  Comma  Common  No  Do  ‘Slight’  ‘Marked’  Insert macro

26.  Saturday, n=33 reports (only Kadom)  Error rate: 16/33 = 48% (Baseline 52%)  Monday, n=53 reports (co-authored only)  Error rate: 15/53 = 28% (Baseline 37%)  Overall error rate: 38% (Baseline 45%)

27.  Process measure  Weekday, 100% of reports were proofread  Despite proofreading, there were some errors remaining in the report before signoff  Balancing measure  15% time spent proofreading  Time spent proofreading each report = 2min (mean)  Macro: Manual insertion takes only 2s longer RVUs do NOT cover VR (used to be covered by technical fee)

28.  Continue to monitor % error in reports  Aim at error rate < 22% Reflections:  I thought I was proof reading, but not focused  I did not know I had to serve the dictation software; I thought the dictation software served me.

30. Galen of PergamonThomas Sydenham Hippocrates

31. 19992001 IOM Reports

36. ResearchQI PurposeDiscover new knowledgeBring knowledge to practice TestsLarge blinded testMany sequential tests BiasesControlStabilize from test to test Data Gather as much as possible Learn just enough for modification DurationLongShort IRB required at BMC if publication considered

39. WALTER A. SHEWHART  March 18, 1891 – March 11, 1967  American physicist, engineer and statistician  Western Electric Company  Importance of reducing variation  Continual process-adjustment  Control chart  “Shewhart cycle” W. EDWARDS DEMING  October 14, 1900 – December 20, 1993  American engineer, statistician, professor, author, lecturer, and management consultant  Plan-Do-Study-Act (PDSA)-cycle  Highly influential on post WWII economic growth in Japan  Toyota’s famous production system (“A3”)

40. Appreciation of a system Understanding the overall processes involving suppliers, producers, and customer/recipients Knowledge of variation Use of statistical sampling to determine range and causes of variation Theory of knowledge The limits of what can be known Knowledge of psychology Concepts of human nature

41.  Langley GL, Moen R, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance (2nd edition). San Francisco: Jossey-Bass Publishers; 2009.The Improvement Guide: A Practical Approach to Enhancing Organizational Performance Forming the Team Setting Aims Establishing Measures Selecting Changes Testing Changes Implementing Changes Spreading Changes } Plan DoStudy Act

44.  Instructions  Pre-Survey  My IHI certificates  Planning Phase  Current state & Data display  PDSA cycles  Summary  Post-survey Assess & Improve ACGME & Great teaching

45.  Instructions  Pre-Survey  My IHI certificates  Planning Phase  Current state & Data display  PDSA cycles  Summary  Post-survey