1. Using Mixed Methods to Produce a Systematic Review of the Literature
Dr. Elizabeth G. Creamer Professor Cherie Edwards, Doctoral Student School of Education, Virginia Tech, Blacksburg, VA
EERA, 2/19/16

2. Definition Systematic Review
A systematic review or research synthesis is a type of literature review that uses systematic procedures to synthesizes the literature in order to pool or aggregate the results or to evaluate quality features.
Mixed methods research synthesis(MMRS)-review includes empirical articles using qualitative, quantitative, and mixed methods
The goal in this type of research is to generalize results across studies and to produce a synthesis that combines words and numbers.
Systematic=replicable
A type of Content Analysis
EERA, 2/19/16

3. Topics from My Fall Class
Sexual Assault Prevention Training- Jeanne Knies
Programs for Veterans – Chris Davidson
Arts Based Programs – Cherie Edwards
Communication Technologies – Sreyoshi Bhaduri
This are about what in the medical field are called “interventions.” In education, these would be about programs/activities/classes where outcomes studies.
Also had projects about:
First Year Engineering Programs
Community Gardens
EERA, 2/19/16

4. QUAL, QUANT, and MM Approaches to a Systematic Review
QUANT Approach
QUANT Articles
Calculates an effect size for outcomes across programs
QUAL Approach
QUAL Articles
Produces a narrative, generally summarizing themes
Sometimes uses case studies of exemplars
MIXED METHOD Approach
QUAL, QUANT, & Mixed Articles
EERA, 2/19/16

5. What is meant by “systematic”?
Replicable “governed by a set of rules that are explicitly stated and applied equally to all units of analysis” (Kripenndorff, 2004, p. 19)
Systematic selection of articles
Systematic screening of articles for quality
Systematic coding procedures
Systematic analytical procedures
EERA, 2/19/16

6. How do journal articles become data?
INDUCTIVE CODING
DEDUCTIVE CODING
What contextual factors are associated with the most effective activities/ programs?
What are the unique characteristics of the most effective programs and the people who participate in them?
Length/intensity of program
Number of participants
Retention rates
Types of outcomes
Cognitive
Affective
Interpersonal/Social
A Mixed Methods Approach would include both inductive and deductive coding.
EERA, 2/19/16

7. GENERIC STAGES FOR A MIXED MEHTOD STUDY
Define the research purpose and questions.
Set parameters for a systematic literature search, including criteria for inclusion and exclusion, and conducting the search.
Screen articles for quality using a rubric.
Code text in the articles using both an inductive and deductive approach.
Identify categories, themes, and/or relationships about results or outcomes using using both an inductive and a deductive approach.
Summarize result by counting, plotting (as through cluster analysis), using figures, or a narrative summary.
EERA, 2/19/16

8. Example of A Rubric for Screening Articles for Quality
Challenge in a mixed method review is to identify a rubric that applies to QUAL, QUANT, and MM articles, without introducing bias in favor of one approach.
EERA, 2/19/16

9. This is from Chris Davidson’s summary of the literature about programs for veterans. He produced a GT model.
EERA, 2/19/16

10. This is a MIXING TABLE

11. Systematic Reviews: Challenges
Getting an adequate sample size (20-40) of articles on the same construct or intervention to conduct statistical procedures.
Developing creative figures/models/visualizations.
Examples of a mixed methods mixed synthesis (MMMR) are few and far between.
Many of the example I have used are from the qualitative research literature. I do not find it difficult to imagine how these can be readily extended to employ a mixed method approach, but others may not find it that that.
EERA, 2/19/16

12. Discussion Questions
Has anyone in the audience had experience with this kind of review?
What support/opposition might there be at your home institution for this kind of research?
EERA, 2/19/16

13. Relevant References
Attride-Stirling, J. (2001). Thematic networks: An analytical tool for qualitative research. Qualitative Research, 1(3), ).
Kearney, M. H. (2001). Enduring love: A grounded formal theory of women’s experience with domestic violence. Research in Nursing & Health, 24,
Onwuegbuzie, A. J., & Teddlie, C. (2003). A framework for analyzing data in mixed methods research. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp ). Thousand Oaks, CA.SAGE.
Sandelowski, Voils, Leeman, & Crandall. (2012). Mapping the mixed methods- mixed research synthesis terrain. Journal of Mixed Methods Research, 6 (4),
Whittemore, R. (2005). Analysis of integration in nursing science and practice. Journal of Nursing Scholarship, 37(3),
EERA, 2/19/16