1. Evidence Synthesis/Systematic Reviews of Eyewitness Accuracy
Joanne Yaffe
9/14/2017
Research supported by the Laura and John Arnold Foundation

2. Overview The Science of Research Synthesis
Procedures of Research Synthesis
General Rationale for Our Research
Phases of Our Research
Overview of Evidence Syntheses in Eyewitness Identification
Scoping Review of System/Estimator Variables and Eyewitness Accuracy
Systematic Reviews of Selected Factors Influencing Eyewitness Identification
Network Meta-Analyses Across Systematic Reviews

3. The Science of Research Synthesis
Explosion of research since the 1960s prompted renewed interest in combining studies.
Traditional literature reviews are “haphazard” in terms of inclusion of studies, “cognitive calculus” for combining. In short, they are “irreproducible.”
Overarching Goals of Systematic Reviews
Limit bias
Enhance methodological rigor
Employ better and more appropriate statistical tools

4. What is a Systematic Review?
Common labels include:
research synthesis,
research review,
systematic review,
quantitative review, and
meta-analysis.
Some scholars use the term “meta-analysis” to refer to only the quantitative summaries used in a systematic review, others use it more broadly.

5. Systematic Review versus Meta-Analysis
A systematic review need not include a meta-analysis.
A meta-analysis need not be based on a systematic review (though often it should be!)

6. A Systematic Review Should Contain . . .
a priori statement of questions and methods,
a clearly defined, explicit question,
a comprehensive, reproducible, and systematic search for studies,
an explicit, reproducible strategy for screening and including studies,
explicit, reproducible data extraction (coding),
systematic examination of the quality of included studies,
appropriate analysis and reporting of results,
interpretations supported by data, and
implications for future research, and if relevant, for policy or practice.

7. General Rationale for Our Research: Why our work is important
Although there are a number of systematic reviews, meta-analyses and other evidence syntheses found in the literature on accuracy of eyewitness identification, the National Research Council Report (2014) examined several reviews and concluded that they were largely methodologically flawed, but there has been no systematic, methodologically sound synthesis.
We are sure that there has been a lot of research on the effects of System and Estimator Variables on the accuracy of eyewitness identification, but we don’t have a systematic mapping of what research is available using what types of studies and on what variables.
Further, we hope to synthesize data on similar studies in a series of systematic reviews to establish more definitive answers and to establish research synthesis methods for the field as well as inform new primary research.

8. Phases of Our Research
Overview of Evidence Syntheses in Eyewitness Identification
Scoping Review of System/Estimator Variables and Eyewitness Accuracy
Systematic Reviews of Selected Factors Influencing Eyewitness Identification (5-8 of these)
Network Meta-Analyses Across Systematic Reviews

9. Overview of Evidence Syntheses in Eyewitness Identification
RATIONALE:
Although there are a number of systematic reviews, meta-analyses and other evidence syntheses found in the literature on accuracy of eyewitness identification, the National Research Council Report (2014) examined several reviews and concluded that they were largely methodologically flawed, but there has been no systematic, methodologically sound synthesis. This overview seeks to remedy this gap.
OBJECTIVES:
Conduct a sensitive and extensive search for evidence syntheses related to eyewitness identification
Assess the quality of these syntheses using a validated instrument, AMSTAR
Describe the methodological guidance followed by authors of the included reviews
Describe the extent to which this body of research conforms to PRISMA Reporting Standards.
Provide an inventory of Estimator and System variables investigated

10. Overview of Evidence Syntheses in Eyewitness Identification
METHODS:
Find all available evidence syntheses of quantitative studies of variables affecting accuracy of eyewitness identification
Extract data related to objectives, methods used, analyses used, including effect size computations, and included studies
EXPECTED OUTCOMES:
Description of existing synthesis methods, the questions addressed, the statistical methods used, the effect sizes calculated and a beginning assessment of what we know
PROGRESS:
Protocol filed on OSF
Search Strategy designed
Search started
Data extraction design started

11. Scoping Review of System/Estimator Variables and Eyewitness Accuracy
RATIONALE:
The National Research Council report (2014) concluded that the state of scientific research on eyewitness identification is unsettled
NRC Report suggested that the field would benefit from:
more effective designs
combining more than one variable at a time
using diverse study populations
using more informative statistical measures and analyses
Using more transparent and reproducible systematic reviews to examine moderators of relationships
In order to plan sophisticated systematic reviews, it would be useful to start with a mapping of the available experimental research.
Scoping studies, increasingly used in reviews of health evidence, “aim to map rapidly the key concepts underpinning a research area and the main sources and types of evidence available, and can be undertaken as stand-alone projects in their own right, especially where an area is complex or has not been reviewed comprehensively before" (Mays, Roberts, & Popay, 2001, p. 194)

12. Scoping Review of System/Estimator Variables and Eyewitness Accuracy
OBJECTIVES:
Conduct a sensitive and extensive search for evidence about eyewitness identification accuracy and the variables which affect it
Map the key concepts underpinning this research area
Map the main sources and types of evidence available
Map the availability of various research designs, populations used as participants, and variables investigated
Provide an inventory of the types of measures of eyewitness accuracy used
Analyze included studies across reviews to determine: the degree of overlap between studies and gaps in the evidence base.
Provide a database of studies for subsequent focused systematic reviews and possible network meta-analyses.

13. Scoping Review of System/Estimator Variables and Eyewitness Accuracy
METHODS:
locate all available research studies that analyze quantitative data on the accuracy of eyewitness identification
catalog these studies in terms of their
research questions/hypotheses
designs
samples
outcomes measured
analytical strategies
EXPECTED OUTCOMES:
Database of all existing quantitative studies of the relationship between systems and estimator variables and eyewitness accuracy
PROGRESS:
Protocol ready to file on OSF
Search Strategy designed and carried out on Medline. More than 150K potentially includable studies identified

14. Systematic Reviews of Selected Factors Influencing Eyewitness Identification
Rationale:
We need reproducible, trustworthy systematic reviews of research in this field
Need to identify what is known and implications for practice
Need to identify important areas for additional research and implications for design and analysis
Objectives:
Conduct rigorous, transparent synthesis in areas identified as important in first two phases of our work

15. Systematic Reviews of Selected Factors Influencing Eyewitness Identification
Procedures:
Identify studies from database
Extract data
Evaluate the Risk of Bias in included studies
Compute pooled effect sizes using state of art analytical methods
Interpret results in light of Risk of Bias in included studies and in review methods
Expected Outcomes:
Implications for practice
Implications for research
Progress:
Aspiration!

16. Network Meta-Analyses Across Systematic Reviews
Rationale:
It is important to understand the relationships between and among variables affecting eyewitness accuracy
Need to identify what is known and implications for practice
Need to identify important areas for additional research and implications for design and analysis
Objectives:
Conduct network meta-analysis, if possible, across the pooled effects between related meta-analyses from our systematic reviews

17. Network Meta-Analyses Across Systematic Reviews
Procedures:
Consider completed meta-analyses from our systematic reviews
Compute interrelationships
Interpret results in light of Risk of Bias in included studies and in individual reviews
This is new method with its own learning curve
Expected Outcomes:
Implications for practice
Implications for research
Progress:
Not sure this will even be possible, but if it is, it will be completely new.

18. Acknowledgments Michelle Fiander Natasha Naylor The “Team”:
Karen Kafadar
Alice Liu
Brandon Garrett
Chad Dodson