Using Mixed Methods to Produce a Systematic Review of the Literature Dr. Elizabeth G. Creamer creamere@vt.edu Professor Cherie Edwards, Doctoral Student School of Education, Virginia Tech, Blacksburg, VA EERA, 2/19/16

Definition Systematic Review A systematic review or research synthesis is a type of literature review that uses systematic procedures to synthesizes the literature in order to pool or aggregate the results or to evaluate quality features. Mixed methods research synthesis(MMRS)-review includes empirical articles using qualitative, quantitative, and mixed methods The goal in this type of research is to generalize results across studies and to produce a synthesis that combines words and numbers. Systematic=replicable A type of Content Analysis EERA, 2/19/16

Topics from My Fall Class Sexual Assault Prevention Training- Jeanne Knies Programs for Veterans – Chris Davidson Arts Based Programs – Cherie Edwards Communication Technologies – Sreyoshi Bhaduri This are about what in the medical field are called “interventions.” In education, these would be about programs/activities/classes where outcomes studies. Also had projects about: First Year Engineering Programs Community Gardens EERA, 2/19/16

QUAL, QUANT, and MM Approaches to a Systematic Review QUANT Approach QUANT Articles Calculates an effect size for outcomes across programs QUAL Approach QUAL Articles Produces a narrative, generally summarizing themes Sometimes uses case studies of exemplars MIXED METHOD Approach QUAL, QUANT, & Mixed Articles EERA, 2/19/16

What is meant by “systematic”? Replicable “governed by a set of rules that are explicitly stated and applied equally to all units of analysis” (Kripenndorff, 2004, p. 19) Systematic selection of articles Systematic screening of articles for quality Systematic coding procedures Systematic analytical procedures EERA, 2/19/16

How do journal articles become data? INDUCTIVE CODING DEDUCTIVE CODING What contextual factors are associated with the most effective activities/ programs? What are the unique characteristics of the most effective programs and the people who participate in them? Length/intensity of program Number of participants Retention rates Types of outcomes Cognitive Affective Interpersonal/Social A Mixed Methods Approach would include both inductive and deductive coding. EERA, 2/19/16

GENERIC STAGES FOR A MIXED MEHTOD STUDY Define the research purpose and questions. Set parameters for a systematic literature search, including criteria for inclusion and exclusion, and conducting the search. Screen articles for quality using a rubric. Code text in the articles using both an inductive and deductive approach. Identify categories, themes, and/or relationships about results or outcomes using using both an inductive and a deductive approach. Summarize result by counting, plotting (as through cluster analysis), using figures, or a narrative summary. EERA, 2/19/16

Example of A Rubric for Screening Articles for Quality Challenge in a mixed method review is to identify a rubric that applies to QUAL, QUANT, and MM articles, without introducing bias in favor of one approach. EERA, 2/19/16

This is from Chris Davidson’s summary of the literature about programs for veterans. He produced a GT model. EERA, 2/19/16

This is a MIXING TABLE

Systematic Reviews: Challenges Getting an adequate sample size (20-40) of articles on the same construct or intervention to conduct statistical procedures. Developing creative figures/models/visualizations. Examples of a mixed methods mixed synthesis (MMMR) are few and far between. Many of the example I have used are from the qualitative research literature. I do not find it difficult to imagine how these can be readily extended to employ a mixed method approach, but others may not find it that that. EERA, 2/19/16

Discussion Questions Has anyone in the audience had experience with this kind of review? What support/opposition might there be at your home institution for this kind of research? EERA, 2/19/16

Relevant References Attride-Stirling, J. (2001). Thematic networks: An analytical tool for qualitative research. Qualitative Research, 1(3), 385-405). Kearney, M. H. (2001). Enduring love: A grounded formal theory of women’s experience with domestic violence. Research in Nursing & Health, 24, 270-282. Onwuegbuzie, A. J., & Teddlie, C. (2003). A framework for analyzing data in mixed methods research. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 351-383). Thousand Oaks, CA.SAGE. Sandelowski, Voils, Leeman, & Crandall. (2012). Mapping the mixed methods- mixed research synthesis terrain. Journal of Mixed Methods Research, 6 (4), 3-17- 221. Whittemore, R. (2005). Analysis of integration in nursing science and practice. Journal of Nursing Scholarship, 37(3), 261-267. EERA, 2/19/16