1. The quality of reporting of Health Informatics evaluation studies Jan Talmon, Elske Ammenwerth, Thom Geven University Maastricht, UMIT

2. Content Background STARE-HI Study design Results Discussion Future prospects

3. Background HISEVAL workshop 2003 –Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the HIS-EVAL workshop in Innsbruck, IJMI, 2004, 479 Poor quality of manuscripts submitted to MI journals –Editor and reviewer perspective Development of STARE-HI –STAtement on the Reporting of Evaluation studies in Health Informatics

4. Background Study questions: –Can STARE-HI be used for the assessment of the quality of evaluation studies in HI –What is the current quality of reporting –What areas are open for improvement

5. STARE-HI Iterative development –Core writing team (JT, EA), active discussants, open comments through publication on Internet: http://iig.umit.at/efmi/http://iig.umit.at/efmi/ There are 12 item categories described, some expanded –Title, abstract, keywords, introduction*, study context*, methods*, results*, discussion*, conclusion, conflict of interest, references, apendices

6. Study Design Hand search of all issues of 2005 of three major MI journals – consensus (JT&EA) –IJMI, JAMIA, MIM Develop a scoring list from STARE-HI (TG) Test usability of scoring form –3 papers assessed by 5 reviewers Apply revised form on all selected papers

7. Results – paper selection 282 papers reviewed on basis of title and abstract 55 selected by JT 37 selected by EA Initial agreement on 32 Final selection 48 papers: –21 IJMI, 23 JAMIA, 4 MIM

8. Results – Scoring form Extract issues from description in STARE- HI –The abstract should preferably be structured and must clearly describe the objective, setting, participants, measures, study design, major results, limitations and conclusions Each issue scored Maximum score/item –For abstract maximum is 9

9. Results – usability of form 5 reviewers –EA, TG, JB, PN, NdK –All familiar with STARE-HI 3 papers –Fitted to various degrees with STARE-HI (TG) Intraclass Correlation Coefficient for each paper –0.82-0.85

10. Results – usability of form Feedback by scorers revealed some problems –Three items of STARE-HI were not clearly described –Twice it was unclear how extensive a description should be –Six issues could appear at any place in the article – affects reliability of scoring

11. Results – quality of reporting Not all 48 papers could be properly assessed: –Some papers were more descriptions rather than evaluation studies –Secondary analyses –Evaluation of an algorithm or a general application (like email) Final analysis on 39 papers –19 IJMI, 20 JAMIA

12. Results – quality of reporting

13. Title67% Abstract69% Keywords42% Introduction63% Study Context54% Methods50% Results41% Discussion65% Conclusion39% Conflict of interest Acknowledgement 24%

14. Results – some observations

15. Keywords –Study design and “evaluation” often missing Methods –Arguments for study design/case selection are often lacking. Description of intervention, study flow, outcome measures all reported <70%

16. Results – some observations Results –Major findings reported, seldom unexpected findings or unexpected events influencing study Conclusions –Sometimes lacking – even no summary statement, impact of findings, recommendations for future research Conflict of interest/acknowledgment –Often missing. Relation authors-system

17. Discussion/conclusion Applicability STARE-HI –YES –Scope of STARE-HI could be improved –Need for more supporting material on STARE- HI Quality of reporting –Completeness: room for improvement –Completeness measure of quality? Pilot study

18. Future developments STARE-HI will be published in the MI literature Seeking for broader support of STARE-HI –EFMI, IMIA, Journal Editors Development of paper with elaboration on the reasons for items in STARE-HI Broader study on the quality of reporting of evaluation studies in Health Informatics