1. Searching Grey Literature: Is the Effort Worth the Yield? AHRQ Annual Meeting Track C, Session 105 Methods for Synthesizing Evidence Marian McDonagh, PharmD Evidence-based Practice Center

2. The Problem: Searching for and including grey literature adds substantial workload to a systematic review. The benefit is unclear in terms of value for time spent Information from grey literature may be limited and risk of bias cannot be determined. 2

3. Session Purpose and Goals This session will explore –The methodological argument in favor of searching for grey literature –Different approaches to searching for grey literature with highest yield –The usefulness of including grey literature based on real-world examples 3

4. Discussion Leaders Dr. David Moher –Ottawa Hospital Research Institute –University of Ottawa –Ottawa Evidence-based Practice Center Rose Relevo, MLIS, MS –Oregon Evidence-based Practice Center –Oregon Health & Science University Marian McDonagh, PharmD –Oregon Evidence-based Practice Center –Oregon Health & Science University

5. Usefulness of Grey Literature What have we discussed so far today: Evidence that failure to include grey literature can lead to biased result (Dr. Moher) –Methods to identify grey literature for comparative effectiveness reviews (Ms. Relevo) Interactive Session: What’s it like in practice? –Examples from the Drug Effectiveness Review Project use of FDA materials as grey literature

6. Drug Effectiveness Review Project DERP is a collaboration of primarily state Medicaid agencies that have joined together to commission systematic reviews of the comparative effectiveness and harms of drugs in many widely used drug classes, and to apply the findings to inform public policy in local settings. Project began in 2003 –OHSU Center for Evidence-based Policy coordinates –EPCs at Oregon and University of North Carolina produce systematic reviews for the project. 6

7. DERP I (15) 2003-2006 DERP II (17) 2006-2009 DERP III (12) 2009 - 2012 Alaska Arkansas CADTH* California HCF Idaho Kansas Michigan Minnesota Missouri Montana North Carolina Oregon Washington Wisconsin Wyoming Alaska Arkansas California California HCF CADTH* Idaho Kansas Michigan Minnesota Missouri Montana New York North Carolina Oregon Washington Wisconsin Wyoming Arkansas CADTH* Colorado Idaho Maryland Missouri Montana New York Oregon Washington Wisconsin Wyoming * Canadian Agency for Drugs and Technologies in Health DERP Organizations

8. 114 Reports Completed to date Proton Pump Inhibitors Long-acting Opioids Statins Non-steroidal Anti-Inflammatory Drugs (NSAIDs) Estrogens Triptans Skeletal Muscle Relaxants Oral Hypoglycemics Drugs for Overactive Bladder ACE Inhibitors Beta Blockers Calcium Channel Blockers Angiotensin II Receptor Antagonists 2 nd Generation Antidepressants Drugs for Constipation Direct Renin Inhibitors/ACEI/ AIIRA Fibromyalgia, Drugs to treat Drugs for Neuropathic Pain Antiepileptic Drugs in Bipolar Disorder/ Pain 2 nd Generation Antihistamines Atypical Antipsychotics Inhaled Corticosteroids ADHD and ADD, Drugs to treat Alzheimer's, Drugs to treat Antiplatelet Drugs Thiazolidinediones (glitazones) Drugs for Hepatitis C Newer Drugs for Insomnia Targeted Immune Modulators Beta Agonists Newer Anti-emetics Drugs for Multiple Sclerosis Combination Products Controller Drugs for Asthma Newer Drugs for Diabetes Topical Calcineurin Inhibitors

9. DERP Methods in Using FDA Documents Systematic review methodology for DERP includes searching FDA web site –FDA Medical and Statistical reviews of NDAs Documents available on FDA web site > 1997 Approved drugs, newly approved for updates Include studies if enough information to rate quality What we are looking for: –Unique unpublished studies –Unpublished data related to published studies

10. Workload? Number of documents reviewed varies Range 0 to 9 FDA documents per report (mean 1.5) Document Size and Organization Issues Several hundred pages each Organization varies Historically pdfs of scanned originals Matching published and unpublished studies Review after electronic searching and selection Dual review: eligibility, quality, abstraction

11. DERP Studying Experience with FDA Documents We are analyzing DERP reports for Frequency of information found in FDA documents being included in a report Value of information from FDA documents included Value is being assessed by assigning an impact category to each instance, e.g. fills a gap in evidence 175 FDA documents eligible for review in114 reports over 9 years –48 instances of evidence from FDA documents included (27%)

12. Example 1: Unique unpublished study Atypical Antipsychotic Drugs (2004) –Head to head comparisons of primary relevance to review –No eligible published studies of aripiprazole (Abilify®) identified through usual searches and request for manufacturer information – Abilify® FDA document found > 200 hundred pages, Examined 5 trials Identified one trial head to head trial (N = 255)

13. Your thoughts? If you came across this evidence what would do with it? 1.Nothing. Don’t use grey literature. 2.Include in report qualitatively only 3.Include in report qualitatively, but influences publication bias assessment 4.Fully include, incorporate into meta- analysis, adjust strength of evidence

14. Your Thoughts? If you had included this information, how do you think it would have changed the SOE? 1.Yes 2.No 3.Unsure

15. Your Thoughts? Do you believe the added information was valuable to the review’s conclusions? 1.Yes 2.No 3.Unsure

16. Your thoughts? Do you think that the time and resources spent to obtain this information was worthwhile? 1.Yes 2.No 3.Unsure

17. Discussion Panelists Thoughts? What we did in DERP –Included study in report; changed conclusions only in a minor way Secondary outcome measure used (cognitive function) Aripiprazole superior to olanzapine on some, but not all measures Further discussion?

18. Example 2: Unpublished data Drugs for Insomnia, Update 2 (2008) Zolpidem ER (Ambien CR ® ) new –3 published placebo-controlled trials identified –5 placebo-controlled trials in FDA documents Matched 2 of the published trials FDA docs: Primary outcome at 8 hours –Ambien superior to placebo up to 6 hours, not different at hours 7 and 8 –Publication: reports only 6-hour results

19. Your thoughts? If you came across this evidence what would do with it? 1.Nothing. Don’t use grey literature. 2.Include in report qualitatively only 3.Include in report qualitatively, but influences publication bias assessment 4.Fully include, incorporate into meta- analysis, adjust strength of evidence

20. Your Thoughts? If you had included this information, how do you think it would have changed the SOE? 1.Yes 2.No 3.Unsure

21. Your Thoughts? Do you believe the added information was valuable to the review’s conclusions? 1.Yes 2.No 3.Unsure

22. Your thoughts? Do you think that the time and resources spent to obtain this information was worthwhile? 1.Yes 2.No 3.Unsure

23. Discussion Panelists Thoughts? What we did in DERP –Included results from FDA docs – changed conclusions of report on Ambien CR® Further discussion?

24. Example 3: Unique unpublished study Proton Pump Inhibitors, Update 2 (2004) Esomeprazole (Nexium®) vs omeprazole –FDA documents: 4 trials submitted prior to the 2001 approval of Nexium® –2 published studies identified Both found esomeprazole superior to omeprazole –2 unpublished studies (in FDA docs) found no statistically significant difference between the drugs

25. Your thoughts? If you came across this evidence what would do with it? 1.Nothing. Don’t use grey literature. 2.Include in report qualitatively only 3.Include in report qualitatively, but influences publication bias assessment 4.Fully include, incorporate into meta- analysis, adjust strength of evidence

26. Your Thoughts? If you had included this information, how do you think it would have changed the SOE? 1.Yes 2.No 3.Unsure

27. Your Thoughts? Do you believe the added information was valuable to the review’s conclusions? 1.Yes 2.No 3.Unsure

28. Your thoughts? Do you think that the time and resources spent to obtain this information was worthwhile? 1.Yes 2.No 3.Unsure

29. Discussion Panelists Thoughts? What we did in DERP –Included study in report only qualitatively –Inadequate information in FDA docs to evaluate quality –Subsequent update: obtained data from manufacturer and incorporated into meta- analysis Further discussion?

30. Wrap up The Problem Revisited –Searching for and including grey literature adds substantial workload to a systematic review. –The benefit is unclear in terms of value for time spent –Information from grey literature may be limited and risk of bias cannot be determined.