Gyte G a, Grant-Pearce C b, Henderson S a, Horey D a, Oliver S c and Sakala C a a Cochrane Pregnancy and Childbirth Group, University of Liverpool (UK); b PREST, University of Manchester; c SSRU, Institute of Education, University of London. Does consumer refereeing improve the quality of systematic reviews of healthcare interventions? Perspective of editors and authors Acknowledgements: This research was supported by a ‘Discretionary Funding’ grant from the Cochrane Collaboration Steering Group Objective: To determine how editors and review authors view consumer refereeing within the editorial process for preparing systematic reviews of effects of healthcare interventions; in particular, their assessment of the impact of consumer involvement on the quality of Cochrane reviews, and lessons for consumer involvement in healthcare research more generally. This information was sought to help plan a more extensive evaluation. Design: An independent researcher undertook semi-structured telephone interviews with editors, review authors, consumers, consumer co-ordinators and the co-ordinator of a Cochrane review group. The researcher examined routine editorial documentation and undertook mapping interviews to understand aims of involving consumers in research and the Cochrane Collaboration ’ s rationale for involving consumers as referees. A short questionnaire, asking for overall views of consumer input into the editorial process, identified review authors and consumers for telephone interview. This presentation reports results from interviews with five review authors selected to give diverse views, along with four editors and the Group ’ s co-ordinator. Consumer views are reported elsewhere. Interviews were transcribed, and the main issues, impressions and themes from each were summarised, with resulting data explored to identify themes. Results: Key points identified were:  quality of consumer input was perceived to be positive;  those with an overview of the review process considered that consumer input improved the final review;  earlier consumer input may be beneficial Conclusions: This evaluation has identified key issues surrounding consumer refereeing of systematic reviews undertaken within the Cochrane Collaboration. Consumers were considered to provide important contributions, and suggestions for improvements in process were made. Further research is planned to assess more specifically what additional contribution consumers make, and whether objectively consumers improve the quality of Cochrane systematic reviews of healthcare interventions. Abstract Abstract

Background: Cochrane Pregnancy and Childbirth Group involves consumers as referees in the production of systematic reviews of healthcare interventions to:  ensure the reviews are targeted at problems important to people  take into account outcomes important to people ensure accessibility to people making decisions  adequately reflect variability in values, conditions & healthcare in different countries The PCG Consumer Panel (PCGCP) refereeing process:  Panel includes part-time paid coordinator, 2 volunteer regional coordinators, 72 volunteer consumers  Consumers choose of the 60 protocols/reviews a year on which they wish to comment  Consumer comments are collated and summarised by the Consumer Panel Coordinator Consumers, in coming from a different perspective, add an important and valued contribution The next phase of the evaluation should proceed, seeking more objective measures of quality Aims: Aims: To determine how editors and review authors view consumer refereeing within the editorial process for preparing systematic reviews of effects of healthcare interventions; in particular, their assessment of the impact of consumer involvement on the quality of Cochrane reviews. This information was sought to help plan a more extensive evaluation. Contact:: Gill Gyte at Acknowledgements: This research was supported by a ‘Discretionary Funding’ grant from the Cochrane Collaboration Steering Group Design:  2 mapping interviews were undertaken to understand aims of consumer involvement  A preliminary questionnaire was sent to all review authors and consumers with the group (around 300), to select for interview. 70 replied  5 authors and 5 consumers chosen to give a range of views. Consumer views are reported elsewhere. 4 editors with the longest experience with the group were chosen  Interview questions were developed by the independent researcher based on an understanding of the review group processes, the aims of involving consumers as referees and the aims of the project  Interviews were transcribed, and the main issues, impressions and themes from each were summarised, with resulting data explored to identify overall themes  18 semi-structured one-hour telephone interviews were undertaken with 4 editors, 5 review authors, 3 consumer coordinators, 5 consumers and the review group coordinator 1. Consumer input was viewed positively, especially by editors 2. Consumers brought added value in the form of a different perspective 3. Consumers provided suggestions for: clarity of language clarity of language more meaningful outcomes more meaningful outcomes more comprehensive enquiry more comprehensive enquiry “So it’s to do with clarity of language, but it’s also to do with the centrality of, the importance of trying to put their views right at the centre, and their perspective.” “…it’s important that the Consumer Panel continues to teach us about the importance of the persons, of the patients’ view, of adverse effects, and of an accessible language” author “I think the summary comments from our consumer coordinator, they are by far the biggest influence” editor “I think I can speak for all [the] editors on this because it comes up at every single one of our Editorial meetings, that the input in general is fantastic. It’s the highest quality we get. Its certainly the most detailed and thoughtful” “…a very important role for the consumers in recommending what issues need to be addressed in future trials” “…it was very long when it came. I think it was about 18 pages…it took me a while to get over the heart-sink. I mean it was enormously valuable editor “… they thought particular outcomes were important, and we hadn’t actually considered those as important outcomes and in fact when we thought about it, they were…..” author Results Results Conclusion author author “…it works well because we’ve got a … a dedicated person, that can co-ordinate and work with consumers, who understands what they are saying, at the same time understands what the review needs.” editor editor