1. Systematic Reviews: 9 Lessons Learned Beth A. Tarini, MD Robert Wood Johnson Clinical Scholar University of Washington July 14, 2005

2. Needed a project for residency research elective Needed a project for residency research elective No other specific research interests at the time No other specific research interests at the time NOTE: Shortly after I began this project, I developed an interest in the role of diagnostic testing in generating harm in pediatric patients NOTE: Shortly after I began this project, I developed an interest in the role of diagnostic testing in generating harm in pediatric patients Why did I do this? Lesson # 1: Be interested (very, if possible) in your topic

3. Question generated from patient encounter during residency: Question generated from patient encounter during residency: Attending told me “what we do”, BUT ALSO… Attending told me “what we do”, BUT ALSO… Challenged the evidence behind the practice Challenged the evidence behind the practice And suggested that I turn the question into a systematic review And suggested that I turn the question into a systematic review How did I chose the topic? Is it OK to give peanut butter to a 12 mo. old?

4. How did I chose my mentors? Attending was too busy mentoring fellows Attending was too busy mentoring fellows RWJ fellow was willing to mentor me RWJ fellow was willing to mentor me Good situation for me Good situation for me I had a lot of questions I had a lot of questions The fellow was accessible & very competent (had done a systematic review before) The fellow was accessible & very competent (had done a systematic review before) Accessibility of mentor may be more important than job title Lesson # 2:

5. Finding a question First, formulated the question First, formulated the question Does early solid feeding in infants increase the risk of allergy? Does early solid feeding in infants increase the risk of allergy? NOTE: I modified it after searching the literature NOTE: I modified it after searching the literature Wanted an interesting question Wanted an interesting question Wanted to appeal to a broad audience Wanted to appeal to a broad audience Find an interesting question that will lead to a publication Lesson # 3:

6. Finding a question Then, refined the question… Then, refined the question… Early – before 4 months of age Early – before 4 months of age Solids – any food stuff other than breastmilk, cow’s milk or formula Solids – any food stuff other than breastmilk, cow’s milk or formula Allergy – any clinically diagnosed allergic disease (initially) Allergy – any clinically diagnosed allergic disease (initially) In the end – asthma, eczema, food allergy, pollen allergy, allergic rhinitis, and animal dander allergy* In the end – asthma, eczema, food allergy, pollen allergy, allergic rhinitis, and animal dander allergy* Lesson # 4: Be Specific

7. The search… Searched databases (MEDLINE, EMBASE, COCHRANE) & bibliographies of relevant studies and review articles Searched databases (MEDLINE, EMBASE, COCHRANE) & bibliographies of relevant studies and review articles Working with the librarians Working with the librarians 1 st SEARCH 1 st SEARCH Searched Medline with MeSH headings ~ 1800 titles Searched Medline with MeSH headings ~ 1800 titles Found multiple “missed” articles during bibliography search Found multiple “missed” articles during bibliography search

8. The search… 2 nd SEARCH 2 nd SEARCH Searched MEDLINE with keywords “infant” and “food” and “allergy” Searched MEDLINE with keywords “infant” and “food” and “allergy” ~ 2500 citations ~ 2500 citations Lesson # 5: Cast a wide net

9. Reviewing the studies Inclusion/Exclusion criteria modified during review Inclusion/Exclusion criteria modified during review Reviewed 39 articles* Reviewed 39 articles* In general, studies of poor quality In general, studies of poor quality No randomized controlled trials No randomized controlled trials Majority were cohort studies Majority were cohort studies Many studies inadequately controlled Many studies inadequately controlled The literature is like a box of chocolates…you never know what you’re gonna get Lesson #6:

10. What did I find? The evidence is of mediocre quality & conflicting The evidence is of mediocre quality & conflicting Some evidence to support an increased risk of eczema with early solid feeding Some evidence to support an increased risk of eczema with early solid feeding Worse for those with a family history of eczema Worse for those with a family history of eczema

11. Budget Didn’t have one – “do as I say, not as I do” Didn’t have one – “do as I say, not as I do” Photocopying and ordering articles was the biggest expense Photocopying and ordering articles was the biggest expense The RWJ fellow gave me photocopy cards The RWJ fellow gave me photocopy cards Children’s orders articles for staff for free Children’s orders articles for staff for free Overall estimate - $1000-1500? Overall estimate - $1000-1500? Lesson #7: You always need some $

12. My “expected” timeline October 2002: Begin research October 2002: Begin research November 2002: Finish research November 2002: Finish research December 2002: Submit for publication December 2002: Submit for publication Clearly, I was new to this… Clearly, I was new to this…

13. The real timeline October ’02: Began research December ’02: Applied to RWJ ….residency cont’d December ’03: Submitted abstract for peds meeting July – Dec ’04: Finished writing manuscript January ’05: Submitted manuscript March ’05: Rejected April ’05: Revised & Resubmitted May ‘05: Manuscript accepted for publication 2002 20032004 2005

14. The real timeline It always takes longer than you think it will Lesson #8:

15. What has this study done for me? Reinforced my skepticism that standard practice is often not based on solid evidence Reinforced my skepticism that standard practice is often not based on solid evidence Taught me the value & necessity of being specific Taught me the value & necessity of being specific Improved my ability to critique studies Improved my ability to critique studies Improved my patience and stamina Improved my patience and stamina Lesson #9: You always learn something