1. Publication Bias in Health Informatics: Results of a survey Nicolette de Keizer Amsterdam, The Netherlands UMIT Elske Ammenwerth Innsbruck, Austria

2. Aim  What percentage of IT evaluation studies does not result in internationally available publications?  What are the reasons for not publishing the results of IT evaluation studies

3. Methods - Survey  E-mail based survey: Which system, which aspect and how did you evaluate? Where was it published What were reasons not to publish (6 predefined categories or free text)  742 email adresses of: EFMI/IMIA WG on assessment of HIS, AMIA WG on evaluation, Pubmed (paper 2001-2006 on evaluation of IT)  136 CIOs

4. Results  Response rate CIOs= 3.7% (5/136)  Response rate= 18.3% (136/742)  141 respondents: 118 valid information on IT evaluations studies 19 no evaluation studies conducted 4 unclear/unusable answers

5. Results Under-representation some non-English- speaking countries and commercial domain

6. Results  217 evaluation studies, 213 publications 103

7. Results Source of publications of 103 evaluation studies

8. Results “The results and systems are too specific to interest audience in other countries” “May publish following validation. Probably not [journal name] which seems to be closed shop for those outside of the academic communities ” “Overwhelmed with ongoing tasks to implement EMR. Always a lack of time when funding is coming from outside.” “The setup (e.g. amount of interviews) was not robust enough.” “The results were not all positive. The system administrators and stakeholders of the system would not like to accept the result.” “The organization I work(ed) in prohibited publication (well, not expressly, but it was made clear that such a publication would not be welcome).” “The evaluations were done for a government agency. Once the reports were delivered, no further activities were funded. The majority of authors were consultants, not academics. Publication is not of interest for consultants.” “Not subjected to our university ethics process, therefore not published.”

9. Discussion of methods  Strong academic bias  English-speaking countries bias  No guarantee that provided information is complete or correct  Results only illustrative, not representative for the IT community in health informatics

10. Discussion of results  Only half of the studies internationally published  Limited number of HI journals or proceedings and large variation of medical journals  Various reasons for non-publication but no strong hints for publication bias due to negative results

11. Recommendations to increase published evaluation studies for EBHI  Raise number of conducted studies Rewards for evaluation, budget, outside IT project team  Raise number of submitted papers GEP-HI  Raise acceptance rate of submissions STARE-HI  Increase internationally available papers Motivate authors to publish, EVALDB (http://evaldb.umit.at )http://evaldb.umit.at