1. Assembling a Systematic Review Team: Balancing Expertise and Potential Conflicts of Interest Avoiding bias in systematic reviews AHRQ annual meeting Bethesda North Marriott Conference Center September 20, 2011

2. IOM recommendations STANDARD 2.1 Establish a team with appropriate expertise and experience to conduct the systematic review 2.1.1 Include expertise in the pertinent clinical content areas 2.1.1 Include expertise in the pertinent clinical content areas STANDARD 2.2 Manage bias and conflict of interest (COI) of the team conducting the systematic review STANDARD 2.2 Manage bias and conflict of interest (COI) of the team conducting the systematic review 2.2.3 Exclude individuals whose professional or intellectual bias would diminish the credibility of the review in the eyes of the intended users 2.2.3 Exclude individuals whose professional or intellectual bias would diminish the credibility of the review in the eyes of the intended users STANDARD 2.4 Manage bias and COI for individuals providing input into the systematic review 2.4.2 Exclude input from individuals whose COI or bias would diminish the credibility of the review in the eyes of the intended users 2.4.2 Exclude input from individuals whose COI or bias would diminish the credibility of the review in the eyes of the intended users STANDARD 2.7 Submit the protocol for peer review 2.7.1 Provide a public comment period for the protocol and publicly report on disposition of comments 2.7.1 Provide a public comment period for the protocol and publicly report on disposition of comments

3. Definitions Bias Bias – a systematic error or deviation from the truth, in results or inferences (Higgins JPT, et al. Cochrane handbook 2009). – inclination to present or hold a partial perspective at the expense of (possibly equally valid) alternatives Conflict of interest (COI) Conflict of interest (COI) – a set of conditions in which professional judgment concerning a primary interest (such as the validity of research) tends to be unduly influenced by a secondary interest (such as financial or professional gain) (Thompson, 1993). Expertise Expertise – Extensive knowledge based on research, experience, or occupation and in a particular area of study. Expert is someone who is a widely recognized authority by peers and public.

4. Potential biases in systematic reviews Holding an established opinion on the answers to the research questions of the review; the inclination to present or hold a partial perspective at the expense of (possibly equally valid) alternatives Holding an established opinion on the answers to the research questions of the review; the inclination to present or hold a partial perspective at the expense of (possibly equally valid) alternatives – Individuals in a particular specialty may have an unbalanced familiarity for one type of intervention or technology. – Individuals livelihood may depend on finding greater benefit of one type of intervention or technology over a competitor. – Experts may hold a previously formed opinion and be unlikely to change. Holding an established opinion on the quality of studies to be included. Holding an established opinion on the quality of studies to be included. – Individuals who conducted a study or trial may not be able to assess the quality of their own or other “competitor” studies in an unbiased manner.

5. My take home points We are concerned with bias, but this is difficult to measure. We are concerned with bias, but this is difficult to measure. Non-financial conflicts of interest can result in bias. Non-financial conflicts of interest can result in bias. – Perceived conflicts of interest may still reduce the credibility of the review in the eyes of the intended users – Conflicts due to familiarity or livelihood based on an intervention or technology can be managed by balancing the team and input from different specialty areas. Expertise can be a form of bias Expertise can be a form of bias – May be difficult to discern unless they have published an editorial or commentary on the subject. – Not including appropriate expertise may reduce the credibility of the review in the eyes of the intended users – It is important to hear opinions from all sides, but the core team should be without bias. Individuals should not be involved in grading their own study, either at the individual study level or at the body of evidence level. Individuals should not be involved in grading their own study, either at the individual study level or at the body of evidence level.

6. Introductions and Acknowledgements Susan Norris, Oregon Health and Science University Susan Norris, Oregon Health and Science University Gillian Sanders Schmidler, Duke University Gillian Sanders Schmidler, Duke University AHRQ staff Elise Berliner Elise Berliner Christine Chang Christine Chang Yen-Pin Chiang Yen-Pin Chiang Supriya Janakiraman Supriya Janakiraman Elisabeth Kato Elisabeth Kato Mary Nix Mary Nix EPC workgroup EPC workgroup Eric Bass Eric Bass Stanley Ip Stanley Ip Melissa Mcpheeters Melissa Mcpheeters Sydney Newberry Sydney Newberry Susan Norris Susan Norris Margaret Piper Margaret Piper Paul Shekelle Paul Shekelle Meera Viswanathan Meera Viswanathan Evelyn Whitlock Evelyn Whitlock Renee Wilson Renee Wilson Michael White Michael White

7. Objectives Provide a framework for thinking about intellectual conflict of interest Provide a framework for thinking about intellectual conflict of interest Review of the existing literature on the potential effects of intellectual conflict of interest Review of the existing literature on the potential effects of intellectual conflict of interest Work through examples of potential conflicts of interest when conducting a systematic review Work through examples of potential conflicts of interest when conducting a systematic review

8. Case Examples

9. Procedure specialist A review is comparing surgical, medical or alternative treatments. Should a specialist who makes a living doing only surgical interventions be the principal investigator? 1. Yes 2. Yes, but with restrictions 3. No 4. It depends

10. Procedure specialist A review is comparing surgical, medical or alternative treatments. Should a specialist who makes a living doing only surgical interventions be part of the core team? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No 5. It depends

11. Professional society staff Should a scientist who works for a specialty professional society be the principal investigator of a review? 1. Yes 2. Yes, but with restrictions 3. No 4. It depends

12. Professional society staff Should a scientist who works for a specialty professional society be a co-author of a review? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No 5. It depends

13. Professional society staff Should a scientist who works for a specialty professional society be on the technical expert panel? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No 5. It depends

14. Professional society staff Should a scientist who works for a specialty professional society be a peer reviewer? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No 5. It depends

15. Active research in area A scientist has federal grants to study one of the intervention of interest. Should this person be the principal investigator of the review? 1. Yes 2. Yes, but with restrictions 3. No 4. It depends

16. Active research in area A scientist has federal grants to study one of the intervention of interest. Should this person be a co-author of the review? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No 5. It depends

17. Active research in area A scientist has federal grants to study one of the intervention of interest. Should this person be on the technical expert panel? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No 5. It depends

18. Active research in area A scientist has federal grants to study one of the intervention of interest. Should this person be a peer reviewer? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No 5. It depends

19. Published A scientist was an author of a study that would potentially be included in the review. Should this person be the principal investigator? 1. Yes 2. Yes, but with restrictions 3. No 4. It depends

20. Published A scientist was an author of a study that would potentially be included in the review. Should this person be on the core team? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No

21. Published A scientist was an author of a study that would potentially be included in the review. Should this person be on the technical expert panel? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No

22. Published A scientist was an author of a study that would potentially be included in the review. Should this person be a peer reviewer? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No 5. It depends

23. Involved in study A scientist coordinated one center of a multi-center trial that would potentially be included in a review. Should this be the principal investigator? 1. Yes 2. Yes, but with restrictions 3. No 4. It depends

24. Involved in study A scientist coordinated one center of a multi-center trial that would potentially be included in a review. Should this be part of the core team? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No 5. It depends

25. Published review or guidelines A scientist has previously published a review or guideline on the subject of the review. Should this person be the principal investigator? 1. Yes 2. Yes, but with restrictions 3. No 4. It depends

26. Published review or guidelines A scientist has previously published a review or guideline on the subject of the review. Should this person be a co-author? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No 5. It depends

27. Scientific advisor or editor A scientist serves on an advisory or editorial board without financial compensation on a related topic to the review. Should this person be the principal investigator? 1. Yes 2. Yes, but with restrictions 3. No 4. It depends

28. Scientific advisor or editor A scientist serves on an advisory or editorial board without financial compensation on a related topic to the review. Should this person be a co- author? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No 5. It depends

29. Scientific advisor or editor A scientist serves on an advisory or editorial board without financial compensation on a related topic to the review. Should this person be on the technical expert panel? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No 5. It depends

30. Scientific advisor or editor A scientist serves on an advisory or editorial board without financial compensation on a related topic to the review. Should this person be a peer reviewer? 1. Yes 2. Yes, but balance with other perspectives 3. Yes, but with restrictions 4. No 5. It depends