Kevin Krost, M.A. & Elizabeth Creamer, Ed.D. Mixed Methods Systematic Literature Review of Mixed Methods Instrument Development and Validation Kevin Krost, M.A. & Elizabeth Creamer, Ed.D. Educational Research and Evaluation (EDRE) Ph.D. program, Virginia Tech, Blacksburg, VA

Background When developing and validating instruments across content areas, the emphasis has traditionally been on quantitative methods as the sole provider of evidence (Daigneault & Jacob, 2014) There has been an increase in the amount of mixed methods research conducted on instrument development and validation, requiring synthesis

Background Validity is largely considered a quantitative concept Different terms used in qualitative research to describe similar phenomena Credibility Transferability Dependability Confirmability

Research Purpose Provide a comprehensive and cohesive analysis of the previous research conducted on instrument development and validation using mixed methods Fill the gap within MMIDV by providing a comprehensive, systematic literature review to describe it

Research Questions Research Question 1 (Mixed): What are mixed methods design characteristics of mixed methods instrument development and validation articles? Research Question 2 (Qualitative): Do the authors identify the value-added of a mixed method? If so, what characteristics are present? Research Question 3 (Quantitative): What design features of mixed methods studies are related to mixed methods evaluation rubric score?

Methods RQ 1 (Mixed): Inductive, emergent open- and focused-coding to derive quality criteria score level meanings. Descriptive statistics and correlations to describe each criterion and their relationships. RQ 2 (Qualitative): Inductive, open-coding to evaluate presence of value-added statement. Focused coding to derive characteristics, or themes. Research Question 3 (Quantitative): Linear and curvilinear regression of criterion scores on overall quality score to evaluate contributions of each.

Mixing Table Article Criteria and Inductively-Derived Score Levels Article Critique Criteria   Amount of Mixing Appropriateness of Analyses Mixed Methods Rationale 1 Mixing During Discussion One Successful Method No Rationale 2 Mixing During Analysis One Adequate and Inadequate Method Unclear Rationale 3 Mixing During Methods Both Methods Adequate Clear Rationale 4 Mixing During Literature Both Methods Exceptional Value-Added Statement Example "a mixed method design was developed that, as described by Tashakkori and Teddlie (1998), includes...” (4) "Specifically, the data were analyzed using principle components analysis (PCA) with varimax rotation” (3) “Notably, although the research was aimed at exploring the supportive care needs of the men and their partners, this paper focuses on…“ (1)

Mixing Table Descriptive Statistics and Correlations (Quant) between each Criterion (Qual) and Total Score   Amount of Mixing Appropriateness of Analyses Mixed Methods Rationale Criteria Total Score Mean 3.000 3.0667 2.7333 8.8000 Median 3 8 Mode 4 12 Standard Deviation .8452 .8837 1.1629 2.624 1 0.6695 0.7268 0.7831 Criteria Total Score 0.8696 0.8994 0.9410

Qualitative Table Value-Added Codes with Illustrative Statements and Corresponding Article Numbers Primary Code Illustrative Statement Article Numbers Deeper understanding of phenomenon "The MR has proven to be a suitable paradigm for studying the causes of DIF and also offering a more integrated view of the phenomenon; it has led to conclusions that could not be obtained by applying a single method" 1, 7, 10 Unique contributions of both methods "The quantitative and qualitative methods in the current study were highly integrated – that is, the whole of the findings exceed the sum of the individual quantitative and qualitative parts" 4, 15

Quantitative Table Category Scores and Composite Score by Article Number Amount of Mixing Appropriateness of Analyses Mixed Methods Rationale Criteria Total Score 1 4 12 2 5 3 7 6 8 9 10 11 13 14 15

Quality Score Variability

Prediction of Quality Score by Criteria

Mixed RQ Results Each criterion was fairly consistently high Sufficient variability indicating distinct, effective score meanings Score meanings based on quantity and adequacy of criteria Rationale lowest and most variable criterion Highest correlation between rationale and appropriateness of analyses Perfect score most common among appropriateness and rationale (Mode = 4)

Qualitative RQ Results Several articles discussed the value-added of using mixed methods Highly related to quality based on mixed RQ Some explicitly stated value-added, others implied it and were analyzed Both codes applied to a couple of articles Each focused-code fairly saturated given n Moderate number of articles given N = 15 Maximum points for rationale, potential for perfect quality score

Quantitative RQ Results Overall quality score quite variable Mode = 12, SD = 2.624 4 out of 5 value-added articles had max score MM rationale contributed most to quality (R2) Indicated the importance of value-added statements to overall quality Could be related to journals in question Average quality score high (M = 8.8, Med =8) Amount of mixing least-related to quality Few “Low” quality articles, more “Medium”

Conclusions There were clear, discrete themes to describe the criterion score levels Quantitative analysis supported this and the importance of rationale and value-added The value-added of conducting MMIDV was captured by two codes both which made sense High R2 values from prediction of overall quality score gave more evidence to support clear MM rationales and value-added

Questions? Thank you!