Evidence Synthesis/Systematic Reviews of Eyewitness Accuracy Joanne Yaffe 9/14/2017 Research supported by the Laura and John Arnold Foundation

Overview The Science of Research Synthesis Procedures of Research Synthesis General Rationale for Our Research Phases of Our Research Overview of Evidence Syntheses in Eyewitness Identification Scoping Review of System/Estimator Variables and Eyewitness Accuracy Systematic Reviews of Selected Factors Influencing Eyewitness Identification Network Meta-Analyses Across Systematic Reviews

The Science of Research Synthesis Explosion of research since the 1960s prompted renewed interest in combining studies. Traditional literature reviews are “haphazard” in terms of inclusion of studies, “cognitive calculus” for combining. In short, they are “irreproducible.” Overarching Goals of Systematic Reviews Limit bias Enhance methodological rigor Employ better and more appropriate statistical tools

What is a Systematic Review? Common labels include: research synthesis, research review, systematic review, quantitative review, and meta-analysis. Some scholars use the term “meta-analysis” to refer to only the quantitative summaries used in a systematic review, others use it more broadly.

Systematic Review versus Meta-Analysis A systematic review need not include a meta-analysis. A meta-analysis need not be based on a systematic review (though often it should be!)

A Systematic Review Should Contain . . . a priori statement of questions and methods, a clearly defined, explicit question, a comprehensive, reproducible, and systematic search for studies, an explicit, reproducible strategy for screening and including studies, explicit, reproducible data extraction (coding), systematic examination of the quality of included studies, appropriate analysis and reporting of results, interpretations supported by data, and implications for future research, and if relevant, for policy or practice.

General Rationale for Our Research: Why our work is important Although there are a number of systematic reviews, meta-analyses and other evidence syntheses found in the literature on accuracy of eyewitness identification, the National Research Council Report (2014) examined several reviews and concluded that they were largely methodologically flawed, but there has been no systematic, methodologically sound synthesis. We are sure that there has been a lot of research on the effects of System and Estimator Variables on the accuracy of eyewitness identification, but we don’t have a systematic mapping of what research is available using what types of studies and on what variables. Further, we hope to synthesize data on similar studies in a series of systematic reviews to establish more definitive answers and to establish research synthesis methods for the field as well as inform new primary research.

Phases of Our Research Overview of Evidence Syntheses in Eyewitness Identification Scoping Review of System/Estimator Variables and Eyewitness Accuracy Systematic Reviews of Selected Factors Influencing Eyewitness Identification (5-8 of these) Network Meta-Analyses Across Systematic Reviews

Overview of Evidence Syntheses in Eyewitness Identification RATIONALE: Although there are a number of systematic reviews, meta-analyses and other evidence syntheses found in the literature on accuracy of eyewitness identification, the National Research Council Report (2014) examined several reviews and concluded that they were largely methodologically flawed, but there has been no systematic, methodologically sound synthesis. This overview seeks to remedy this gap. OBJECTIVES: Conduct a sensitive and extensive search for evidence syntheses related to eyewitness identification Assess the quality of these syntheses using a validated instrument, AMSTAR Describe the methodological guidance followed by authors of the included reviews Describe the extent to which this body of research conforms to PRISMA Reporting Standards. Provide an inventory of Estimator and System variables investigated

Overview of Evidence Syntheses in Eyewitness Identification METHODS: Find all available evidence syntheses of quantitative studies of variables affecting accuracy of eyewitness identification Extract data related to objectives, methods used, analyses used, including effect size computations, and included studies EXPECTED OUTCOMES: Description of existing synthesis methods, the questions addressed, the statistical methods used, the effect sizes calculated and a beginning assessment of what we know PROGRESS: Protocol filed on OSF Search Strategy designed Search started Data extraction design started

Scoping Review of System/Estimator Variables and Eyewitness Accuracy RATIONALE: The National Research Council report (2014) concluded that the state of scientific research on eyewitness identification is unsettled NRC Report suggested that the field would benefit from: more effective designs combining more than one variable at a time using diverse study populations using more informative statistical measures and analyses Using more transparent and reproducible systematic reviews to examine moderators of relationships In order to plan sophisticated systematic reviews, it would be useful to start with a mapping of the available experimental research. Scoping studies, increasingly used in reviews of health evidence, “aim to map rapidly the key concepts underpinning a research area and the main sources and types of evidence available, and can be undertaken as stand-alone projects in their own right, especially where an area is complex or has not been reviewed comprehensively before" (Mays, Roberts, & Popay, 2001, p. 194)

Scoping Review of System/Estimator Variables and Eyewitness Accuracy OBJECTIVES: Conduct a sensitive and extensive search for evidence about eyewitness identification accuracy and the variables which affect it Map the key concepts underpinning this research area Map the main sources and types of evidence available Map the availability of various research designs, populations used as participants, and variables investigated Provide an inventory of the types of measures of eyewitness accuracy used Analyze included studies across reviews to determine: the degree of overlap between studies and gaps in the evidence base. Provide a database of studies for subsequent focused systematic reviews and possible network meta-analyses.

Scoping Review of System/Estimator Variables and Eyewitness Accuracy METHODS: locate all available research studies that analyze quantitative data on the accuracy of eyewitness identification catalog these studies in terms of their research questions/hypotheses designs samples outcomes measured analytical strategies EXPECTED OUTCOMES: Database of all existing quantitative studies of the relationship between systems and estimator variables and eyewitness accuracy PROGRESS: Protocol ready to file on OSF Search Strategy designed and carried out on Medline. More than 150K potentially includable studies identified

Systematic Reviews of Selected Factors Influencing Eyewitness Identification Rationale: We need reproducible, trustworthy systematic reviews of research in this field Need to identify what is known and implications for practice Need to identify important areas for additional research and implications for design and analysis Objectives: Conduct rigorous, transparent synthesis in areas identified as important in first two phases of our work

Systematic Reviews of Selected Factors Influencing Eyewitness Identification Procedures: Identify studies from database Extract data Evaluate the Risk of Bias in included studies Compute pooled effect sizes using state of art analytical methods Interpret results in light of Risk of Bias in included studies and in review methods Expected Outcomes: Implications for practice Implications for research Progress: Aspiration!

Network Meta-Analyses Across Systematic Reviews Rationale: It is important to understand the relationships between and among variables affecting eyewitness accuracy Need to identify what is known and implications for practice Need to identify important areas for additional research and implications for design and analysis Objectives: Conduct network meta-analysis, if possible, across the pooled effects between related meta-analyses from our systematic reviews

Network Meta-Analyses Across Systematic Reviews Procedures: Consider completed meta-analyses from our systematic reviews Compute interrelationships Interpret results in light of Risk of Bias in included studies and in individual reviews This is new method with its own learning curve Expected Outcomes: Implications for practice Implications for research Progress: Not sure this will even be possible, but if it is, it will be completely new.

Acknowledgments Michelle Fiander Natasha Naylor The “Team”: Karen Kafadar Alice Liu Brandon Garrett Chad Dodson