What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information Unit 1

This session Evidence reviews: The what, why and how… Evidence reviews in practice: Examples of recent Rapid Evidence Assessments

Evidence reviews: the what, why and how… Why review evidence? “The sheer amount of potential research evidence in most substantive areas of social science and public policy… make[s] it almost impossible to keep abreast of the research literature in any one area”. Davies, 2003 The first step in developing evidence based policy and practice is to review existing material to establish what is already known. Need to know what we know Need to identify gaps in the evidence base and prioritise these based on risk Policing as a profession needs to undertake the same journey as medicine…

Methods for reviewing evidence… Review type Systematic? Timeframe Literature Review No 1-8 weeks Scoping Review Rapid Evidence Assessment Yes 2-6 months Systematic Review 8-12+ months Multi-arm systematic review 12+ months Review of Reviews Variable The first step in developing evidence based policy and practice is to review existing material to establish what is already known.

Literature reviews Look! The breadcrumbs lead here, this MUST be the answer!

Literature reviews AHAHAHAHA! I have tricked you into only reviewing only *some* of the available evidence!

Literature reviews Advantages: Provide information on a specific topic in a very short period of time. A suitable method when time, money and access to support are limited. Disadvantages: Prone to selection and publication bias: they only tend to review evidence that is readily available they can be over reliant on sources that disproportionately report studies with positive outcomes. Lack of clarity on methodology - they do not state: reasons for the inclusion or exclusion of literature/ evidence how literature has been appraised/ assessed for quality how conclusions have been reached. Advantages: Literature reviews can be useful for providing information on a specific topic in a very short period of time. It is feasible to carry out a literature review alone and is therefore a suitable method when time, money and access to external support are limited. Disadvantages: Literature reviews collate studies that are relevant to a particular topic and summarise and appraise the research in order to draw conclusions from it. However, literature reviews do not explicitly set out how the studies will be found, included and analysed; their findings should therefore be treated with a great deal of caution.

Are we happy for professionals to only have some of the evidence when making decisions?

We need to look at all the evidence… Antman et al, 1992. Study comparing recommendations for treating heart attacks based on literature reviews with recommendations based on a systematic meta-analysis. Literature reviews often failed to mention important advances or exhibited delays in recommending effective preventive measures. The literature review was the dominant model until the late 1980s, when, in the health field, a number of studies were published which showed inadequacies and bias in the process. One study, for example, showed that successful treatment for heart attacks would have been identified earlier if systematic review methods had been used to summarise knowledge (Antman et al 1992). Illustrating that literature reviews are weak when the review question aims to synthesis findings on the outcomes from specific interventions In some cases, treatments that have no effect on mortality or are potentially harmful continued to be recommended by several clinical experts.

Systematic Reviews

Systematic reviews Overall, legitimacy interventions resulted in a large, significant increase in positive perceptions of police. Taken from: Mazerolle, L., Bennett, S., Davis, J., Sargeant, E. and Manning, M. (2013) Procedural Justice and Police Legitimacy: A Systematic Review of the Research Evidence. Campbell Systematic Reviews 2013:1.

Systematic reviews Advantages: Minimise the bias in the way studies are found, included and synthesised. Transparent approach allows future studies to be added to the review Enables a cumulative body of sound evidence to be developed on a subject area over time. It should be possible for anyone else to conduct the same review and come to the same conclusions. Disadvantages: Systematic reviews are as comprehensive as possible and are therefore very time consuming. They typically require a team of researchers (either ‘in-house’ or contracted to carry out the work) and are therefore resource intensive. Systematic reviews have explicit objectives and studies are chosen on explicit criteria. A thorough search for studies is conducted using electronic and print sources and the grey literature (unpublished/ work in progress) as well as hand searching journals and textbooks, searching of specialist websites, and use of personal contacts. Each study found is screened according to uniform criteria and the reasons for excluding studies clearly documented (Cochrane Collaboration 2007).

Policy opinions of systematic reviews Of those that had heard of them, some had concerns: Timeliness Relevance Usefulness Some had difficulty distinguishing them from literature reviews, even when explained Campbell S et al (2007) Analysis for policy: evidence-based policy in practice TIMELINESS There was an awareness that systematic reviews do take some time There were some reservations that they could be done to a policy timetable RELEVANCE Relates to timeliness – if it is not available when needed, is it of any use at all? Will it give the evidence needed? MANAGING EXPECTATIONS. USEFULNESS Linked to the above two, unless it is relevant and timely, it will not be useful. Some of those interviewed who said they did know what systematic reviews were but then gave descriptions of literature reviews. So their opinions related more to literature reviews.

Rapid Evidence Assessments Use systematic principles and the same process as a systematic review… …but make compromises given available time and resources Pragmatic and transparent approach Advantages: Rapid Evidence Assessments can be undertaken in the following circumstances: When there is uncertainty about the effectiveness of a policy or service and there has been some previous research. When a policy decision is required within months and policy makers/researchers want to make decisions based on the best available evidence within that time. At policy development stage, when evidence of the likely effects of an intervention is required. When it is known that there is a wide range of research on a subject but questions still remain unanswered. When a map of evidence in a topic area is required to determine whether there is any existing evidence and to direct future research needs. As a starting point. Ideally, one is undertaken to answer a particularly pressing policy concern, and once the immediate question is answered it can form the basis of a more detailed full systematic review. In such cases, a Rapid Evidence Assessment could be better described as an ‘interim evidence assessment’. In these situations an REA can provide a quick synthesis of the available evidence by shortening the traditional systematic review process. Disadvantages? In shortening the traditional systematic review process REAs risk introducing bias. Systematic reviews also suffer from biases but limiting the process increases the risk of them occurring. For example, limiting the search to published literature may introduce bias into the REA because of the exclusion of unpublished material. Therefore, the need for the evidence to be provided rapidly should outweigh the risk of increased bias. REAs (and all other review methods) should record how they have been less comprehensive than a full systematic review and discuss the likely levels of bias this has caused so that those taking decisions are aware of limitations of the evidence. All review methods, including REAs, risk generating inconclusive findings that provide a weak answer to the original question. For example, there may not be studies of sufficient methodological quality to address the question. The tight timescales in an REA mean that if findings are inconclusive there is less time than in a systematic review to go back and reformulate the question or inclusion criteria. Terms Anywhere in the article “Domestic violence” 597,000 “Domestic abuse” 32,500 Title only “Domestic violence“ 23,700 “Domestic abuse” 1,220 Adding ‘systematic review’ “Domestic violence“ 7 “Domestic abuse” 0

Rapid Evidence Assessments REAs provide a balanced assessment of what is already known about a policy or practice issue, by using systematic review methods to search and critically appraise existing research. They aim to be rigorous and explicit in method and thus systematic but make concessions to the breadth or depth of the process by limiting particular aspects of the systematic review process. The speed at which this is undertaken will depend on how quickly the evidence is needed, the available resource to carry out the REA and the extent to which reviewers are prepared to limit the systematic review process. i.e. you can compromise, as long as you explain how…

The process (in a nutshell) Draft search terms Draft sift criteria Sift received abstracts Request relevant papers Read and ‘grade’ papers Write it up (‘synthesis’) Be clear that these will be illustrated with examples below…

Some key principles Demonstrate consistency in searching/ sifting Document search and sift process Process should be transparent and repeatable Specify required quality of evidence Systematic reviews only? Pre-post studies only? All ‘empirical’ papers? Be explicit/ transparent about the limitations of the approach So what are these systematic principles? Broadly, there are four things you need to focus on. Demonstrate the consistency of your searching a sifting. Make sure that you write up and quality assure the search criteria and sift criteria before you start searching or sifting. By all means pilot the criteria on one database/ a few papers. But…. Make sure you document any changes or adaptations, and how you have addressed any particular problems or issues in the searching/ sifting process. In the example we are discussing, an additional ‘’sift criteria’ needed to be added as a lot of papers on changing the way doctors treated particular illnesses were included after the sift stage. Specifying the quality of evidence required for inclusion is key. In this example, we included only reviews and or pre-post studies, this is probably the most important factor in limiting your search to manageable numbers. The quality of evidence you require is going to be linked to the question your customer has, or the kind of statements you want to be able to make as a result of the review. There will always be limitations to a review. This is absolutely nothing to be worried about or to try to hide. Be explicit and clear about the caveats and qualifications involved in the process. (Your customers should already understand some of the limitations).

Some examples What it is like in practice?

One practical example Review of Police Leadership and Training commissioned by the Home Secretary CC Peter Neyroud needed the best evidence he could get on “What works in training and behaviour change?” Three weeks to deliver an evidence review Not a full REA, but… More than literature or scoping reviews Used systematic principles Due to time limit, search restricted to evaluations and systematic reviews only Context: Had to produce rapid evidence reviews in three weeks to inform the Neyroud Review of Police leadership and training. There was another review on leadership which was undertaken by a colleague. Obviously in this sort of situation you need to be mindful of the burden you are putting on the people undertaking searches for you. In our case this was the NPL. These were not full REAs in the sense that we did not include standalone papers. We did reviews of reviews, meaning that when we drafted our search criteria, it had to specify that we only wanted to look at systematic reviews. One note of caution on this is that people often describe reviews that use systematic principles as a systematic review. This is not always the case. The thing that usually differentiates a review using systematic principles from a full review is that there is a more explicit evidential threshold for inclusion in a systematic review.

Searching (Training) Search terms training OR learning OR development AND evaluat* OR assess* OR what works OR impact AND systematic review Limitations Searched 11 databases and 2 websites English language only Evaluation Systematic review These were our search terms… Run through slide… Retrieved papers themselves refer to other papers, but we did not have the time to go to original papers. This is something else that you might do in a longer systematic review, for example. Initial search identified 1,015 abstracts to sift

Available from NPL/ BL in time for inclusion Sifting (Training) Q1: Is the study about adult training, learning, or development? Q2: Is the study: An evaluation (at least pre & post level) OR a systematic review? Secondary sift to remove papers related to inappropriate populations and specific medical conditions Original references 1,015 After first sift Less duplicates After second sift Available from NPL/ BL in time for inclusion 38 32 22 Run through slide… These numbers relate to the training search only. The first sift identified a number of papers which were about training for very specific medical procedures, etc, so these had to be sifted out. Other papers focussing on irrelevant adult populations (e.g. people in drug rehab, etc.) were also removed. This issue would have been avoided (probably) if we had time to test our search terms. Having to make this sort of adaptation to your sifting criteria is okay – as long as you are open and transparent about the change and consistent in applying the new criteria. In total we requested 22 of the 1,015 original references. 10

Read, Appraise and Synthesise papers – What works in training? What works (Good practice) Strong evidence in healthcare shows training that is integrated into routine practice is more effective at changing individual’s attitudes and behaviour than traditional classroom based approaches. Strong evidence from education shows that collaborative continuous professional development is effective in improving pupil outcomes (learning and behaviour) and the practice, attitudes and beliefs of teachers What doesn’t work (Bad practice) There is systematic review evidence that classroom-based training alone is not necessarily an effective way to improve practitioner’s skills or to change their behaviour What’s promising (Promising/ noteworthy practice) There is some evidence that simulation-based training may have some advantage over more traditional classroom methods What’s unknown There is a lack of clear evidence on the effects of reflective practice; portfolio learning; problem based learning; and learning technologies and virtual learning. So our goal at the end of the review was for the reader to have to do as little interpretation of the evidence as possible… When evidence is summarised systematically (now standard in social research), to allow an assessment of ‘what works’ to address a particular policy or practice issues, only studies at level 3 and above with statistically significant results are likely to be included..

Limitations Time Availability of papers (10/22) Unable to pilot search terms Available databases Only those available to the NPL English language only Literature focussed in different areas Almost nothing on policing Papers mostly from healthcare The main limitations for the training review were (1) above all TIME… this meant that we had to be very strict about the papers we included in terms of their availability. Only electronically available papers could be included, as well as anything the British Library could supply in a very quick turnaround. Any thoughts on how there might be a potential bias in this? From my experience, older and more obscure papers are less likely to be electronically available. The other limitations are there for you to see… we could only use databases that were already available to the NPL. And finally, as the process got underway it was very clear that there was next to no literature on training and behaviour change in the policing arena. The papers we identified were mostly from the healthcare area. Any ideas why this might be the case? Level of evidence required has a big impact here. Healthcare/ medicine are good at doing ‘what works’ type evaluations. They understand analysis and evaluation a lot better than many other areas, for example, RCTs and experimental interventions are frequently used for testing drugs and treatments.

Organisational change and business improvement Forces are adopting business improvement techniques to examine current practices and explore scope to change processes to release savings Techniques include QUEST, CI, Lean, Six Sigma, Kaizen, etc. So – are these techniques the answer? Is that magic potion? Or is it snake oil? This is interesting to consider as it is not as simple a question to address as the previous example on training and behavioural change…

Rapid Evidence Assessment Step 1 Systematic search (11,960 abstracts) Step 2 Quality assess & critically appraise (181 empirical studies) Step 3 Synthesise findings (41 studies with useful findings) Draw conclusions Example of the two searches in the organisational change and business improvement REA

Organisational change and business improvement Question Approach Abstracts identified Included papers What works in delivering organisational change? Rapid review of reviews 797 5 What works in organisational change and business improvement? REA 11,163 36 Total of approx 12,000 abstracts; 181 full papers were requested; 41 were included… The red boxes involved using three discrete but similar searches to tackle what is essentially a very broad question. The three searches looked at 1. OC, 2. BI, and 3. specific approaches and factors – e.g. Lean, Kaizen, etc. Why so many papers? Better search terms needed? Dodgy descriptions of papers in abstracts? Problems with searching?

Organisational change – what works? Potential success factors for organisational change Leadership (and having a clear strategy) Resources (financial, personnel and training) Organisational culture and structure Staff feelings of active participation (and related employee empowerment and strong teamwork) Communication …and prior experience of implementing a quality improvement programme

Organisational change: potential success factors Leadership stability of supervision throughout implementation direct support from supervisors - ‘on-the job’ training staff involvement in decisions transformational leadership behaviour = reduced employee cynicism Engagement staff active participation in decision making & ‘room to experiment’ degree to which staff understood rationale for change communication found to influence self reports of job performance

Lessons learned... Piloting your searches is critical (we’ll do this tomorrow) to give you an idea of the size of the job, and if there is much available material It’s important to focus on the end result and how will the findings be used – make it relevant… Recognise (and accept) there are limitations of the approach Make sure you are researching the right question Follow the key principles: Demonstrate consistency in searching/ sifting Document search and sift process Specify required quality of evidence Be explicit about the limitations of the approach REAs should be replicable Run through slide… Explain what you mean by second sifting… Second sift if you have the time: this seems time consuming but makes things quicker in the long run…

The process (for Evidence Base Camp) Stage Who When Draft search terms Review teams EBC Day 2 Draft sift criteria EBC Day 3 (Nov) Sift received abstracts Request full papers NPL and researchers After EBC Day 3 (Nov) Read and grade full papers EBC Day 4/5 (Mar) Synthesise evidence

Search terms are hugely important. Results Anywhere in the article “Domestic violence” 597,000 “Domestic abuse” 32,500 Title only “Domestic violence" 23,700 1,220 Published since 2000 (title only) 13,200 822 Adding ‘randomised controlled trial’ Adding ‘experiment’ 32 Adding ‘systematic review’ 7 Search terms are hugely important. Tomorrow is all about developing search terms and seeing the impact of using different terms. We will be running live pilot searches in your groups with College librarians.

Useful links Civil service REA toolkit: http://www.civilservice.gov.uk/networks/gsr/resources-and-guidance/rapid-evidence-assessment Campbell Collaboration (social interventions, e.g. crime and justice) http://www.campbellcollaboration.org/ Cochrane Collaboration (medical interventions) http://www.cochrane.org/ EPPI centre (education) http://eppi.ioe.ac.uk/cms/