1. CIDER 2015 Conference February 5, 2015 Blacksburg, VA
Conducting a Mixed Method Research Synthesis (MMRS)
Dr. Elizabeth G. Creamer, Professor, School of Education, VA Tech
Dr. Denise R. Simmons, Ass’t Professor, Myers Lawson School of Construction and Engineering Education, VA Tech
Rongrong Yu, Doctoral Student, EDRE, School of Education, VA Tech
CIDER 2015 Conference February 5, 2015 Blacksburg, VA
Creamer, Simmons & Yu, CIDER2015

2. Overview of Topics Covered
1. Describe types and purposes of systematic reviews (SR) of the literature. 2. Review stages in the process a SR. 3. Distinguish QUAL, QUANT, and MM reviews 4. Provide an example of a MMRS 5. Summarize the value-added by a mixed method approach. 6. Identify challenges to systematic reviews 7. Suggest avenues for future research.
Creamer, Simmons & Yu, CIDER2015

3. Defining Mixed Methods Research: Mixing at Any of 4 Stages
QUAL
QUANT MM
Research Question/Design
Data
Sampling or
Analysis
Synthesis or
Visualization
This is not entirely a conventional definition. Most people simply define it at the integration of qualitative and quantitative approaches.
Creamer, Simmons & Yu, CIDER2015

4. How Much of the Research Literature in Education is Mixed Methods?
Discipline
Research Studies
(n=2381)
MM Studies
(n=332 (14%))
Literacy
424
66 (16%)
Mathematics
617
86 (14%)
Science
1182
167 (14%)
Social Studies
158
13 (8%)
19 journals
Reference: Truscott et al., 2010
Creamer, Simmons & Yu, CIDER2015

5. Systematic Literature Reviews – 4 Design Choices
Type of data-numbers or words
Ways to visualize – words, numbers or both
Articles to
Review-QUAL, QUANT, and/or Mixed
Type of analytical procedures-QUAL, QUANT, or both
This is pointing out that there are three design choices:
DATA – do I create a database with words, numbers, or both?
ANALYTICAL PROCEDURES – Do I use qualitative or quantitative procedures, or a combination?
SAMPLING – Do I select to review only QUAL articles? Only QUANT articles? QUANT and QUAL
Creamer, Simmons & Yu, CIDER2015

6. Definition – Research Synthesis
A systematic review or research synthesis is a type of literature review that uses systematic procedures to synthesizes the literature in order to pool or aggregate the results or to evaluate quality features.
Mixed methods research synthesis(MMRS)-review includes empirical articles using qualitative, quantitative, and mixed methods
The goal in this type of research is to generalize results across studies and to produce a synthesis that combines words and numbers.
What is meant by systematic?
Systematic procedures for coding texts, much like you would do a transcript from an interview or focus group.
Creamer, Simmons & Yu, CIDER2015

7. Different Terms Distinguish QUAL, QUANT, and MIXED Research Synthesis
Analytical Logic
Descriptors
QUALITATIVE
Research Synthesis, Meta Narrative, Meta Ethnography, Case Narrative
QUANTITATIVE
Meta-synthesis
MIXED METHODS
Mixed method research synthesis
Mixed method mixed research synthesis*
Integrative mixed method review
Three methodological approaches: qualitative, quantitative, or mixed. The term, meta-synthesis, only applies to a specific kind of QUANT study; The term META SYNTHESIS signals a qualitative study.
We are going to focus on using mixed methods.
Creamer, Simmons & Yu, CIDER2015

8. GENERIC STAGES
Sandelowski et al., (2012, p. 320) identified the following generic steps in conducting a synthetic review.
Define the research purpose and questions.
Set parameters for a systematic literature search, including criteria for inclusion and exclusion, and conducting the search.
Identify categories, themes, and/or relationships about results or outcomes using using either an inductive or deductive approach.
Summarize result by counting, plotting (as through cluster analysis), using figures, or a narrative summary.
Creamer, Simmons & Yu, CIDER2015

9. Sampling in Systematic Reviews
Both purposeful (QUAL) and exhaustive (QUANT)
Goal: articles for a mixed method review
Sampling- goal breadth and depth
Creamer, Simmons & Yu, CIDER2015

10. Value Added of a MM Approach to an Outcome Oriented Review
VERY persuasive foundation for a grant proposal or publication.
Offers the potential to compare outcomes for different groups of students and by type of activity.
Can be used to answer how and why questions and to summarize the literature as a causal model.
It is important to think of these reviews as not just being about triangulation and confirmation.
Creamer, Simmons & Yu, CIDER2015

11. STEP 1 of ANALYSIS: Data Reduction
The first step in most analysis is to create a data base in Excel. This would include both QUAL and QUANT data.
This is a table from an article by Kearney. Kearney used a grounded theory approach to analyze 13 articles reporting on qualitative studies.
This table is an example of the type of quantitative data that can be extracted from an article and added to a qualitative data based.
Kearney, M. H. (2001). Enduring love: A grounded formal theory of women’s experience with domestic violence. Research in Nursing & Health, 24,
Creamer, Simmons & Yu, CIDER2015

12. Data Matrix - Another Type of Data Consolidation
Theme 1
Theme 2
Theme 3
Article 1
XXX XX
Article 2
XXXX
Article 3 X
Article 4
This type of matrix lends itself to counting, with multiple Xs reflecting greater intensity.
The themes could be related to OUTCOMES
This is an excellent way to summarize major trends in the literature.
Creamer, Simmons & Yu, CIDER2015

13. LAST STEP: VISUAL/GRAPHIC DISPLAY – A GT LOGIC DIAGRAM
Systematic reviews can be used to develop, test, or refine theory. This is a data display that reflects a qualitative synthesis.
EXAMPLE – qualitative or quantitative data, qualitative analysis, qualitative synthesis
Example of coding literature in ways that produce a theoretical model for further testing or to drive instrument development.
Whittemore, R. (2005). Analysis of integration in nursing science and practice. Journal of Nursing Scholarship, 37(3),
Creamer, Simmons & Yu, CIDER2015

14. ANOTHER EXAMPLE OF A VISUAL DISPLAY – A CONCEPT MAP
This is another visual display reflecting a QUAL synthesis; like a concept map that summarizes themes and subthemes in a body of literature. In this case, the author only examined qualitative articles on the same construct (intimate partner violence) but the same qualitative approach could be applied to analyze the conclusions of QUANT, QUAL, and mixed articles.
From: Attride-Stirling, 2001, p. 388
Creamer, Simmons & Yu, CIDER2015

15. CAREER: Investigating Co-Curricular Participation of Students Underrepresented in Engineering, Dr. Denise Simmons, PI
The authors would like to gratefully acknowledge the National Science Foundation for their support of this work under the CAREER grant #EEC Any opinions, findings, conclusions, or recommendations expressed in this presentation are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Creamer, Simmons & Yu, CIDER2015

16. Background of Project
Specific outcomes result from involvement in “out of class” experiences
Engineering students are less likely to participate
Few studies investigate
participation decisions of engineering related to these experiences
various types activities when investigating student outcomes
Develop survey to collect data and investigate gaps
Mixed Method Research Synthesis
plays important roles in students’ cognitive and affective development, indicates educational effectiveness, improves educational satisfaction and sense of belonging and can challenge the inequities of the social order by equipping a diverse population with academic and professional skills necessary to succeed in the technological workforce. Yet, undergraduate engineering students are less likely than students in other majors to become involved in co-curricular activities and few studies in engineering education explore multiple outcomes and activities.
Creamer, Simmons & Yu, CIDER2015

17. Research Questions for Co-Curricular Involvement Project
Who benefits from participation in co-curricular activities?
What factors influence students’ decision to participate or not participate in the co-curricular activities?
What are the outcomes of participating in the co-curricular activities?
Does the impact of co-curricular activities vary by types of activities?
How does co-curricular activities influence students’ learning outcomes, persistence in their majors, and career intentions?
Creamer, Simmons & Yu, CIDER2015

18. Conceptual Framework for Mixed Method Research Synthesis
Four steps:
information extraction,
inductive analysis,
cluster analysis, and
analysis of outcomes
Creamer, Simmons & Yu, CIDER2015

19. Steps
Data collection: identified 51 articles with 36 quantitative studies, 7 qualitative studies, and 8 mixed methods studies.
Information extraction: (1) identified the outcomes related to co-curricular involvement; and (2) extracted the information using NVIVO 10.
Creamer, Simmons & Yu, CIDER2015

20. Steps Data analysis—inductive (qualitative) phase
generated descriptive codes through a process of open coding
organized the initial codes into groups and developed categories with a higher level of abstraction
Creamer, Simmons & Yu, CIDER2015

21. Data Consolidation & Comparison
(1) Cluster analysis: grouping categories into clusters by the words similarity
Creamer, Simmons & Yu, CIDER2015

22. Steps Data analysis—deductive (quantitative) phase
(2) Quantizing the results: sort and count the clusters by types of co-curricular activities
Creamer, Simmons & Yu, CIDER2015

23. Systematic Reviews: Challenges
There’s no way you can use this method if you are squeamish about associating numbers with qualitative data or if you see qualitative and quantitative methods as polar opposites.
Getting an adequate sample size (20-40) of articles on the same construct or intervention to conduct statistical procedures.
Developing creative figures/models/visualizations.
Examples of a mixed methods mixed synthesis (MMMR) are few and far between.
Many of the example I have used are from the qualitative research literature. I do not find it difficult to imagine how these can be readily extended to employ a mixed method approach, but others may not find it that that.
Creamer, Simmons & Yu, CIDER2015

24. Discussion Questions: Innovative Approaches to Research Synthesis
What applicability do you see from what we have talked about to your own research?
How could you do a systematic review AFTER you’ve completed a traditional literature review chapter for your chapter?
Creamer, Simmons & Yu, CIDER2015

25. Making Your Mark in Systematic Reviews
A Systematic Review Squared (MRS-squared)
Develop a grounded theory from a review of the literature about how and why an outcome like engagement or interest in STEM is promoted.
Use a QUAL approach to identify the methodological and graphic characteristics of top-cited articles (ie. More than 50 cites in 5 years) in your field
Or reviews published in the Review of Educational Research
Or the visualizations (e.g. figures and tables) presented in these
MRS Square- review of reviews!
(There’s a student directed journal, Educational Review, which might be a less competitive publication venue)
Creamer, Simmons & Yu, CIDER2015

26. Relevant References
Attride-Stirling, J. (2001). Thematic networks: An analytical tool for qualitative research. Qualitative Research, 1(3), ).
Kearney, M. H. (2001). Enduring love: A grounded formal theory of women’s experience with domestic violence. Research in Nursing & Health, 24,
Onwuegbuzie, A. J., & Teddlie, C. (2003). A framework for analyzing data in mixed methods research. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp ). Thousand Oaks, CA.SAGE.
Sandelowski, Voils, Leeman, & Crandall. (2012). Mapping the mixed methods- mixed research synthesis terrain. Journal of Mixed Methods Research, 6 (4),
Whittemore, R. (2005). Analysis of integration in nursing science and practice. Journal of Nursing Scholarship, 37(3),
Creamer, Simmons & Yu, CIDER2015