1. From sifting to synthesis
Evidence Base Camp: February 2014
Levin Wheller
Practice Development Team
Research Analysis and Information Unit 1

2. This session… Recap on REAs While you were away…
Peer review of sifting and the task ahead
Mapping research evidence
Systematic Mapping – the what and the why
An example: Mapping Evidence on Public Order Tactics
Appraising research evidence
The Maryland Scale of Scientific Methods
Some hypothetical examples
Beyond Maryland: non-impact questions
Qualitative methods
Completing a pro forma
Practical group work

3. Recap

4. Systematic reviews and “what works”
Explicit objective/ question
Studies are selected on explicit criteria
Thorough search for studies
Each study is screened according to uniform criteria
Reasons for excluding studies are clearly documented
Advantages:
Systematic reviews are the most robust method for reviewing evidence because they reduce the bias in the way studies are found, included and synthesised. Transparency allows future studies to be added to the review which enables a cumulative body of sound evidence to be developed on a subject area over time. Furthermore, it should be possible for anyone else to conduct the same review and come to the same conclusions.
There is a myth that systematic reviews are only suitable for experimental studies. While it is true that many of the existing systematic reviews have concentrated on large scale quantitative studies, systematic review methods can be applied to any type of question. Where quantitative studies are included it may be possible to undertake a meta-analysis – a method of analysis which combines statistically the results of a number of studies. If more qualitative data are used then a qualitative synthesis can summarise these results. Some reviews do both, and integrate a meta-analysis of data from controlled trials with a synthesis of findings from qualitative studies (Gough & Elbourne, 2002). This is also true for Rapid Evidence Assessments.
Disadvantages:
The strength of systematic reviews is that they aim to be as comprehensive as possible in the studies they find, include and synthesise. However, this leads to their principle disadvantage, which is that they are time consuming. They also require a team of researchers (either ‘in-house’ or contracted to carry out the work) and are therefore resource intensive (although likely to be less so than undertaking a piece of primary research).

5. Rapid Evidence Assessments
Use systematic principles and the same process as a systematic review…
…but make compromises given available time and resources
Pragmatic and transparent approach
Terms
Anywhere in the article
“Domestic violence” 597,000
“Domestic abuse” 32,500
Title only
“Domestic violence“ 23,700
“Domestic abuse” 1,220
Adding ‘systematic review’
“Domestic violence“ 7
“Domestic abuse” 0

6. Some key principles Demonstrate consistency in searching/ sifting
Document search and sift process
Process should be transparent and repeatable
Specify required quality of evidence
Systematic reviews only?
Pre-post studies only?
All ‘empirical’ papers?
Be explicit/ transparent about the limitations of the approach
So what are these systematic principles? Broadly, there are four things you need to focus on.
Demonstrate the consistency of your searching a sifting. Make sure that you write up and quality assure the search criteria and sift criteria before you start searching or sifting. By all means pilot the criteria on one database/ a few papers. But….
Make sure you document any changes or adaptations, and how you have addressed any particular problems or issues in the searching/ sifting process. In the example we are discussing, an additional ‘’sift criteria’ needed to be added as a lot of papers on changing the way doctors treated particular illnesses were included after the sift stage.
Specifying the quality of evidence required for inclusion is key. In this example, we included only reviews and or pre-post studies, this is probably the most important factor in limiting your search to manageable numbers. The quality of evidence you require is going to be linked to the question your customer has, or the kind of statements you want to be able to make as a result of the review.
There will always be limitations to a review. This is absolutely nothing to be worried about or to try to hide. Be explicit and clear about the caveats and qualifications involved in the process. (Your customers should already understand some of the limitations).

7. The process (in a nutshell)
Draft search terms
Draft sift criteria
Sift received abstracts
Request full papers
Read and ‘grade’ papers
Write it up (‘synthesis’)

8. Searching Identifying potentially relevant literature
Sifting Identifying truly relevant literature
Synthesis Summarising what relevant literature tells us

9. Searching Identifying potentially relevant literature

10. The Basics Boolean operators
Boolean is a powerful tool and can be used to create quite complex searches. The standard Boolean operators are:
AND OR
NOT (or AND NOT) (advanced users only!)
Boolean is most effective when used in combination with:
PARENTHESES (round brackets)
TRUNCATION e.g. politic*
WILDCARDS e.g. randomi?e
DOUBLE QUOTES for specific phrases

11. Domestic violence search
Search terms
((((Domestic OR Familial OR Marital OR Spousal OR Partner) Near/2 (Abus* OR violen* OR control* OR coerc*)) OR "domestic homicide" OR "marital rape" OR "intimate partner violence" OR (coercive NEAR/2 (control OR behavio*r*)))
AND (risk AND (predict* OR indicat* OR factor* OR management OR identif* OR assess* OR prevent* OR likelihood OR probability))
AND ("systematic review" OR "literature review*" OR trial* OR "RCT" OR experiment* OR evaluat* OR "best practice*" OR "good practice*" OR effective* OR assess* OR "What Works" OR impact* OR success*))
DA/ DV c 75k papers
‘What works’ c 2m papers
Risk c 1m papers
These were our search terms…
Run through slide…
Retrieved papers themselves refer to other papers, but we did not have the time to go to original papers. This is something else that you might do in a longer systematic review, for example.
Initial search identified 1,531 abstracts to sift

12. Searching as a ‘blunt tool’
Research Question:
What interventions have been shown to be effective in preventing or reducing theft from the person?
So how do we end up with an abstract titled…
Profile on circadian blood pressure and the influencing factors in essential hypertensive patients after treatment
Theft from person synonyms include: Robbery, Mugging, Dipping, Pick pocketing, Assault with intent to rob, Snatching, Theft from person, Bag theft, Bag snatch, Personal theft…

13. Returned by theft search…
Yuan Gang Qiu et al. Profile on circadian blood pressure and the influencing factors in essential hypertensive patients after treatment
Non-dippers were defined as those whose nocturnal decrease in mean systolic BP and/or mean diastolic BP was < 10% of the daytime BP. Binary logistic regression analysis was used to evaluate the correlation between circadian blood pressure profile and factors as gender, age, height, body mass index (BMI), family history of premature cardiovascular disease, women under age 65 or men under age 55, smoking habits, grade of hypertension, and strategy of antihypertensive drugs.
Similarly, the incidence of non-dippers in patients of overweight (24 </= BMI < 28) and obesity (BMI >/= 28) were 3.0 and 4.8 times of those in subjects of normal weight (P = and 0.009, respectively). Compared with patients treated with long-acting calcium channel blockers (CCBs), patients treated with angiotensin-converting enzyme (ACE) inhibitors or angiotensin II receptor blockers (ARBs) alone had less prevalence of nondippers (OR = 0.139, P = 0.010).

14. Sifting Identifying truly relevant literature

15. Sift criteria Question Answer Action Q1. No Exclude Yes Go to Q2
Does the paper consider theft from the person or relevant synonyms? No
Exclude
Yes
Go to Q2
Unclear
Q2.
Does the paper consider interventions that aim to reduce/prevent theft from the person or relevant synonyms?
Go to Q3
Q3.
Does the paper include empirical data or methods?
Include
Can’t exclude
Sift criteria are sometimes called inclusion criteria.
Abstracts are summaries of full papers or books.
The same set of questions are applied to all abstracts to attempt to remove bias/ overt ‘judgement calls’.
Typically, a lot of papers (up to 95%!) identified by the search are excluded at this stage of the process.

16. While you were away…

17. Peer Review: Second sift…
‘Red’ abstracts
‘Dip sample’ taken, e.g. one in 10 ‘red’ abstracts were peer reviewed
Check for ‘false-exclusions’
‘Green’ and ‘Amber’ abstracts
All resifted to review against the inclusion criteria
Check for ‘false-positives’
Decide where possible on ‘Ambers’
Final list of included studies sent to library…
Why do we undertake a second sift?
Basic quality assurance and good practice…
Consistency is important in REAs
Transparent about our inclusion criteria which are
With large groups sifting quickly, we are likely to see differences in interpretation, even using with same sift criteria 17

18. Second sift…
REA
Abstracts
First sift
Second sift
Domestic Abuse
2,678
622
411
Public Order
572
182 38
Always expect to reduce the number of studies substantially through second sifting
Standard process for all REAs and SRs
Dual coding and often used and disagreements discussed between coders to make a final decision
Clarify queries/ refine criteria as a group
Speed of EBC does not enable all of this!!!

19. Available for synthesis
While you were away…
Area
Abstracts returned
Included after 1st sift
Included after 2nd sift
Available for synthesis
Acute Mental Health Crisis
1,408
173 55 38
Barriers to career progression
1,205
153 82 46
Reducing theft from the person
844
192 58
Supporting mental well-being
1,229
162 93 72
Responding to prostitution
979
195 87 48
A final ‘master list’ of included studies was sent to the library for each group
Librarians get hold of papers from the library catalogue or the British Library (if required)
Some studies not available, some not available in English – some were large books which we have excluded due to time constraints…
Also a consideration of cost - £10 per paper we didn’t already have in the library!
Leaves us with a final number ‘available for synthesis’ by you all over the next day… 19

20. Synthesis Summarising what relevant literature tells us

21. Systematic Mapping… Methodology developed by EPPI-Centre
Useful first step of synthesis process
Mapping can:
Describe the nature and coverage of research in the topic area
Identify gaps in the research literature
Describe the design of studies and direct further review work
SCIE Systematic mapping guidance - April 2009
The methodology for systematic mapping was originally developed by the EPPI-Centre
Response to large quantities of evidence which does not measure impact/ is not empirical – review w/ 8000 abstracts, 150 relevant studies, only 6 w/ outcome measures
So mapping a broad area can help direct further research in the future
Typically used in research the College does to look at how to narrow the focus of reviews, e.g. might decide to only look at experimental research, or other reviews, etc. 21

22. Systematic Mapping… What does it involve:
Collection of key information about each study into a pro forma/ template
All have an example in front of you…
This is from Public Order Tactics Review… 22

23. Systematic Mapping…
Collate pro forma results into a searchable bibliographic database
Systematically mapping a topic enables the development of a comprehensive database of literature that includes, but is not necessarily limited to, empirical studies 23

24. What is the evidence on Public Order Tactics? Country of focus24

25. What is the evidence on Public Order Tactics? Research Method25

26. Public Order Tactics - Mapped
Overall: There is only limited research available on public order policing tactics. Only 38 papers were found to be relevant and based on empirical methods.
Age: Most of the research (32/38 papers) has been published since 2000.
Methods: Most papers used qualitative research methods, typically involving observations (n=19) or interviews (n=18). Only one study used experimental methods.
Mapping to existing Authorised Professional Practice Tactics 26

27. Public Order Tactics - Mapped
Unit of study: Only five studies focussed directly on public order tactics. Instead, studies primarily focussed on an event or series of events (n=17) or the police more generally (n=10).
Tactics: Of the 30 approved public order tactics, only nine were mentioned in the research evidence identified for this map. The most frequently mentioned tactics were containment (n=5), Police Liaison Teams (n=3) and CS smoke (n=3).
Mapping to existing Authorised Professional Practice Tactics 27

28. Appraising research evidence

29. Maryland Scientific Methods Scale (SMS). (AKA Maryland Scale)
Developed by Sherman et al (1997)
Established method for judging the quality of impact studies
Useful for judging how much weight to give different studies identified in a ‘what works’/ ‘what’s effective’ REA
Limited use for judging qualitative studies and questions asking ‘how’ things work 29

30. What does ‘good’ or ‘robust’ ‘what works’ evidence look like?
Systematic Reviews
(Based on level 3-5 studies) 5
Randomised controlled trials 4
Before/after measures
Multiple site comparisons 3
Two site comparisons 2
No comparison site 1
One-off measure
Statements about
‘what works’
Statements about ‘what’s promising’
Study designs increasingly rule out potential alternative causes
Should be familiar to you all now…
Study designs cannot rule out potential alternative causes
Statements about possible impact

31. ‘Level 1’ Evidence
“Two months after the introduction of mobile information, officers were spending 55% of their time out on patrol”
Problems?
Can’t rule out other explanations
Time spent on patrol before mobile info?
Is 55% good/ bad/ average?
No before/ after measures
No comparison group
Fictitious example!!!
One off measure with no comparison site…
Can’t say if mobile information makes a difference/ is worthwhile

32. ‘Level 3’ evidence
% of time on patrol
Pre
Post
Western District
(receives intervention)
43%
50%
Eastern District
(no intervention)
46%
We still can’t be sure that improvements are down to the intervention rather than other factors
“Two months after the introduction of mobile information, officers in the Western District had increased their time on the street by 7 percentage points, while officers in the Eastern District had seen an increase of 4 percentage points…”
Need to say this as a statement…
A good study will collect information about processes/ other factors in both areas…
Before/ after measure with 2 site comparison
Problems?
Still can’t rule out other explanations for change
Can’t control for differences between the two areas – different culture, different case load, management, etc.

33. RCTs - ‘gold standard’ evaluation design
2,000 officers
in force
300 officers
in treatment
group
Outcomes
measured
Get
mobile
data
A random
sample of
600 officers
300 officers
in control
group
Outcomes
measured
Business as
usual
In here we can use the example from medicine about the use of steroids for head injuries – how it was received wisdom that these would be useful as steroids reduce swelling, but that in a randomised trial it was shown that there were higher rates of death in those given steroids. The findings were so overwhelming that use of steroids was harmful, that the trial was stopped part way through – a very rare occurrence.
In terms of research methods, this is as good as it gets…
Randomisation controls for differences between groups (e.g. age, gender, length of service) the ensuring differences between treatment and control groups are down to the intervention

34. ‘Level 5’ Evidence - RCTs
“Two months after the start of the trial, officers with access to mobile data were spending 55% of their time out on patrol, whilst officers in the control group were spending 43% of their time out on patrol”
1. Participants are randomly selected from the population and randomly assigned to each group to minimise the chance of bias
2. Randomisation helps us rule out other potential explanations for differences in outcome measures
3. A control group means we can compare differences in outcomes between groups where the only difference is the intervention (e.g. mobile data
Fictitious example!!!
Make sure to be strong in clarifying ‘in this context’ –
Context leaves us with same issue as with our level 3 example – if this mobile data study is (for example) in GMP then there may be things about that specific force that might not be replicated elsewhere… so we *should* look for confirmation that the intervention works in other contexts/ locations before we roll it out everywhere… that’s why we do SYSTEMATIC REVIEWS…
4. RCTs enable us to make strong statements about cause and effect
We can say that officers with access to mobile data spend more time on the street… in this context.

35. Systematic reviews
Overall, legitimacy interventions resulted in a large, significant increase in positive perceptions of police.
MEASURE NUMBER OF DIFFERENT STUDIES, ACCOUNT FOR RANGE OF DIFFERENT CONTEXTS, JUDGE OVERALL EFFECT
SR’s allow us to combine multiple examples of interventions across a range of contexts… and by being systematic in our reviewing, we can ID on average if something is effective generally.
The bigger the sample, the more confidence we can have in its findings.
So – run through…
Taken from: Mazerolle, L., Bennett, S., Davis, J., Sargeant, E. and Manning, M. (2013) Procedural Justice and Police Legitimacy: A Systematic Review of the Research Evidence. Campbell Systematic Reviews 2013:1.

36. What does ‘good’ or ‘robust’ ‘what works’ evidence look like?
Systematic Reviews
(Based on level 3-5 studies) 5
Randomised controlled trials 4
Before/after measures
Multiple site comparisons 3
Two site comparisons 2
No comparison site 1
One-off measure
Statements about
‘what works’
Statements about ‘what’s promising’
Study designs increasingly rule out potential alternative causes
Study designs cannot rule out potential alternative causes
Statements about possible impact

37. Beyond Maryland…
The Maryland scale is focussed on impact questions… and studies might answer a range of other questions…
Question type
Example
Needs
What do people want or need?
Process
Why/how does it work?
Implementation
What is required to make it work?
Correlation
What relationships are seen between phenomena?
Attitudinal
What do people think? What are their experiences?
Economic
How much does it cost and with what benefit/harm?
LINK SOME OF THIS BACK INTO MOBILE DATA EXAMPLE – “if you think about introducing an intervention in your force…”
All of these sorts of questions in the table **should** be part of a high quality experiment
Qualitative studies better at getting in-depth analysis of specific issues
Quantitative studies better at making broad generalisations and measuring impact
How ‘good’ a study is will depend upon how appropriate the methods used are for answering the research question…
Qualitative studies don’t fit into the Maryland dichotomies
You can have different quality qualitative work as well

38. Quality of Qualitative Research
Frameworks to assess/ appraise the quality of qualitative research use ‘principles’ rather than hard-and-fast rules
Cabinet Office e.g. in 2003 sets key principles being that qualitative research should be:
• contributory in advancing wider knowledge
• defensible in design
• rigorous in conduct
• credible in claim
Guiding principles 18 questions over five pages…
There are four central principles that underpin the content of the framework. All of these are based on themes that are highly recurrent in the literature and in the interviews conducted for the study. They advise that research should be:
contributory in advancing wider knowledge or understanding about policy, practice, theory or a particular substantive field;
• defensible in design by providing a research strategy that can address the evaluative questions posed;
• rigorous in conduct through the systematic and transparent collection, analysis and interpretation of qualitative data;
• credible in claim through offering well-founded and plausible arguments about the significance of the evidence generated.
These principles are presented at a sufficiently high level of abstraction that they would apply to a diversity of qualitative approaches. Indeed, most of them are simply emblems of sound and logical enquiry, whatever its form or purpose.

39. What we are doing at EBC Mapping the evidence in our 5 areas…
Presenting key info in charts
Drawing initial conclusions from studies in each area
Some studies will not be impact studies
Presentation of findings to NPLs and senior staff across the service…

40. Completing a pro forma 1. Basic info should be recorded
Reference (title/ author)
Date of publication
Country
2. Check if the study should be included
Check the full paper against the inclusion criteria
If the study meets the inclusion criteria, complete the rest of the pro forma
If not, exclude the study and note your reasons why
Info required
Try looking first in…
Aims
Background/ Introduction section
Methods
Methods section
Quality of evidence
Findings
Discussion/ Results section
Conclusions
Results/ Conclusions section
THE TABLE AIMS TO GIVE SOME SHORTCUTS SO THAT YOU DON’T HAVE TO READ THE FULL PAPER TO GET SOME OF THE KEY INFO 40

41. Synthesis in practice Complete a pro forma for 2 articles
1. GMP study summary – 15 mins
2. An included article for your area – 15 mins
Look at the methods of the study and try to ‘score’ it on the Maryland Scale
Ignore the section on inclusion criteria
Cover findings/ conclusions if you have time but the methods are your focus
Some examples might not fit the scale… 41

42. Synthesis in practice GMP study summary
10 mins to individually complete as much of the pro forma as possible
5 mins to discuss with neighbour/ compare findings
Group feedback 42

43. GMP study summary (1/2) Training
The GMP procedural justice training experiment: Wheller et al
(2013) UK
Policing
Test the impact of training on officer attitudes, officer behaviour and victim satisfaction
Training
576
Random 43

44. GMP study summary (2/2)
Training had a positive impact on officer attitudes, officer behaviour and victim satisfaction
Training works (in this context!) 44

45. Synthesis in practice Included study from your area
10 mins to individually complete as much of the pro forma as possible
5 mins to discuss with neighbour/ compare findings
Group feedback 45

46. Theft study summary (1/2)
Does Heroin Prescription Reduce Crime?: Killias & Rabasa
1997
Policing/ Health
Switzerland
Study of the impact of heroin prescription programme on crime rates in Switzerland
Drug prescription/ treatment
319
Non-random 46

47. Theft study summary (2/2)
Number of outcomes – comparison of rates of (self reported) crime pre and post intervention.
Heroin prescription reduces (self reported) criminality. Police statistics also show reductions in crime. 47

48. Wellbeing study summary (1/2)
Mental, physical, and behavioural outcomes associated with perceived work stress in police officers: Gershon et al
2009
USA
Policing
Study links between perceived stress and impact of ‘coping’ on work stress and health
None – non-impact study
1,072
Non- random 48

49. Wellbeing study summary (2/2)
Outlines a number of statistical relationships drawn from the survey, e.g. between job stressors and work stress
“results underscore the need to re-evaluate police training of recruits… to ensure they get training necessary to meet the daily challenges and demands of police work” 49

50. Career progression study summary (1/2)
Encounters in social cyberspace: e-mentoring for professional women: Headlam-Wells et al
2006 UK
Training
Look at barriers for women in career progression and evaluate the effects of an e-mentoring programme
E-mentoring programme
122 (89)
Non-random 50

51. Career progression study summary (2/2)
Mentoring had a positive impact on mentee ‘employability skills’. Mentoring helped with mentee development.
Suggestion that mentoring may be particularly important to women (but this study does not compare to a similar group of men…) 51

52. Prostitution study summary (1/2)
Attitudes regarding criminal justice responses to sex trafficking: Muftic ?
Bosnia and Herzegovina
Policing
Examine law enforcement attitudes to sanctioning people involved in the sex trade.
None – non-impact study
363
Self reporting 52

53. Prostitution study summary (2/2)
A number of factors are associated with officer support for arrest, etc of trafficking victims, traffickers and customers – including their degree of adherence to prostitution myths
Training for officers “need[s] to include the provision of factual information regarding prostitution and sex trafficking” 53

54. MH crisis study summary (1/2)
Studying Chicago Police Dept’s Crisis Intervention Team: Watson
2010
USA
Policing
Test the impact of training on officer attitudes, officer behaviour and victim satisfaction
Crisis Intervention Team
216
Matched sites 54

55. MH crisis study summary (2/2)
CIT officers directed a significantly greater proportion of subjects to services than non-CIT officers. CIT did not have an impact on arrest decisions.
Challenging to conduct studies of CIT, but possible. Study suggests CIT may improve outcomes of police encounters with persons with mental illness. 55

56. Available for synthesis
What’s next…
Area
Available for synthesis
Acute Mental Health Crisis 38
Barriers to career progression 46
Reducing theft from the person 58
Supporting mental well-being 72
Responding to prostitution 48
Get synthesising!
In the breakout rooms you’ll have everything you need to turn the available papers for your group into a resplendent and wonderful presentation to wow our senior guests tomorrow…
Good luck!
A final ‘master list’ of included studies was sent to the library for each group
Librarians get hold of papers from the library catalogue or the British Library (if required)
Some studies not available, some not available in English – some were large books which we have excluded due to time constraints…
Also a consideration of cost - £10 per paper we didn’t already have in the library!
Leaves us with a final number ‘available for synthesis’ by you all over the next day… 56