1. 1 Assessing quality in systematic reviews of the effectiveness of health promotion and public health (HP/PH): Areas of consensus and dissension Dr Jonathan Shepherd Principal Research Fellow Southampton Health Technology Assessments Centre (SHTAC), School of Medicine, University of Southampton, UK

2. 2 Objectives of today’s presentation  Present results of a series of interviews with systematic reviewers in HP/PH Challenges faced by systematic reviewers Challenges faced by systematic reviewers Barriers / facilitators around assessing quality Barriers / facilitators around assessing quality Barriers / facilitators around learning to do systematic reviews of HP/PH Barriers / facilitators around learning to do systematic reviews of HP/PH

3. 3 Methods  Stage 1 – detailed methodological mapping of random sample of HP/PH systematic reviews (n=30)  Stage 2 – Semi-structured interviews with a sample of systematic reviewers in HP/PH (n=17)

4. 4 Methods (2)  Purposive (n=17)  Cochrane Colloquium (Melbourne, 2005) (n=10/17 interviews)  UK / telephone (2006) (n=7/17 interviews)  Content analysis approach to categorise data into themes  Data analysed using NVivo (2.0)

5. 5 Methods (3) : Interviewees Interviewee’s country Number (%) interviewed Australia 4 (26) Canada 2 (12) Nigeria 1 (6) South Africa 1 (6) United Kingdom 8 (47) USA 1 (6)

6. 6 Results (1): challenges to doing systematic reviews CategoryChallenge Number (%) of interviewees External factors Lack of resources and time 11 (65) Analysis and methods Analysing complex interventions 5 (29) Locating the evidence 4 (24) Analysing observational studies 3 (18) Critical appraisal 3 (18) Dealing with heterogeneity and statistics 3 (18) Developing the review question 2 (12) Lack of flexibility 1 (6) Evidence limitations Reporting bias 2 (12) Poor evidence 1 (6) Expertise Team working 1 (6) Lack of specialist knowledge 1 (6) Miscellaneous Lack of motivation + tedium 1 (6) Advisory group 1 (6) Interpretation and presentation 1 (6) Supporting external groups 1 (6)

7. 7 Results (2): barriers to, and facilitators of, quality assessment Facilitating factors Number (%) interviewees Well written-up studies 9 (53) Good critical appraisal criteria / instrument 9 (53) RCTs / experimental studies 3 (18) Good guidelines 2 (12) Support Consensus on criteria 1 (6)

8. 8 Results (3): Barriers to, and facilitators of, quality assessment (cont) “I mean it is difficult in, you know that there are judgement calls all the time, it isn’t you know it, none of these things are nice easy tick boxes and there’s so many judgements right the way through” “I’ve never met a set of guidelines where they completely dispense with the idea that you make a new judgement” Interviewee 14

9. 9 Results (4): Consensus on quality assessment criteria?  Majority thought there was generally no consensus…  …but more consensus for RCTs  Concept of “Multiple consensus”  Realistic to achieve consensus in HP/PH?

10. 10 Results (4) consensus on quality assessment criteria (cont.) “I’m not sure that that matters that much because people will always do things in different ways. And the idea of forcing an orthodoxy in an area which is incredibly diverse as public health systematic reviews. I mean, I think imagination and lateral thinking bring just as much other help as systematic methods” Interviewee 13

11. 11 Results (5) Learning to do systematic reviews Method of learning Number (%) of interviewees ‘Hands-on’ practice 11 (65) From colleagues and mentors 11 (65) Training courses 10 (59) Literature and written resources 9 (53) Applied existing research skills 5 (29) Academic course 3 (18) Supervising and teaching others 2 (12)

12. 12 Results (6): Challenges in helping others to do reviews Issues Number (%) of interviewees Meta-analysis 7 (54) Quality assessment The need to think criticallyThe need to think critically Using judgmentUsing judgment Conceptually challengingConceptually challenging 4 (31) Principles of evidence-based health 2 (15) Reading and synthesising evidence 2 (15) Study designs and hierarchy of evidence 1 (8) Qualitative data 1 (8)

13. 13 Conclusions: key overarching themes 1. Complexity of many HP/PH reviews

14. 14 “It’s big and diverse, that’s why it’s so much easy just to stick with randomised control trials and be done with it. Or even if you do community trials, cluster randomised, still it’s easier. I think once you move beyond it gets very difficult” Interviewee 4 Conclusions: key overarching themes (cont)

15. 15 Conclusions: key overarching themes (cont) 1. Complexity of many HP/PH reviews 2. Time and resources 3. Subjectivity

16. 16 Recommendations  Adequate planning and resourcing of complex HP/PH systematic reviews  Better reporting of HP/PH studies  Training for critical appraisal opportunities for ‘hands on’ learning opportunities for ‘hands on’ learning acknowledging subjectivity acknowledging subjectivity

17. 17 Acknowledgements  University of Southampton  Professor Katherine Weare  Interviewees who took part in the research  EPPI-Centre, London  Cochrane Public Health Group email: jps@soton.ac.uk