1. How to Design a Mixed Methods Study
by John W. Creswell, Ph.D. and Vicki L. Plano Clark, M.S.
University of Nebraska-Lincoln
Andrews University, July, 2004

3. How would you combine two types of data?
Qualitative Text Data
This is a sample of a text file of words that might be collected on transcripts through interviews, fieldnotes from observations, or from optically-scanned documents.
Quantitative Numeric Data

4. Objectives of the workshop:
Let’s design a mixed methods study
Let’s study how people learn mixed methods research in this room? (or you can work on your own project and follow along at each step)
Let’s start with a title. Write a title.
What data will we collect?

5. What are types of quantitative and qualitative data?
Quantitative data
Close-ended scales
Attitudinal/behavioral scales
Behavioral checklists
Census, attendance records
Qualitative data
Open-ended responses
Semi-structured interviews
Semi-structured observations
Records/documents
Videotapes

6. Let’s identify our quantitative and qualitative data collection

7. Now let’s consider some reasons for why we are collecting (and mixing) both forms of data
Together quantitative and qualitative data provide both precise measurement and generalizability of quantitative research and the in-depth, complex picture of qualitative research
To validate quantitative results with qualitative data
We do not have an adequate instrument. Thus, we need to explore views and develop an instrument
Our quantitative data provide a general explanation and we need to follow-up with participants and have them explain the quantitative results
In our experiment, outcomes to be measured are not enough; they need to be complemented by understanding the process of participants

8. Let’s identify our reason for mixing

9. So… There are good reasons for gathering both forms of data
But…there are certain requirements for this to work best

10. Requirement #1: Now let’s consider whether we have the skills, time, and resources?
We need minimum skills in both qualitative and quantitative data collection. What do we need?
We need time and resources for extensive data collection and analysis. How much time and resources do we need?

11. Write down the skills, time, and resources we will need

12. Requirement #2: The audience(s)
Does our audience appreciate both numbers and stories?
Are they familiar with this design?
Do they need to be educated?
Are examples of published studies available in our content area?

13. Let’s identify the audiences

14. But audiences may not recognize it yet because it is so new
Increased use and acceptance of qualitative research from 1990’s to present
The complexity of our research problems today requires understanding trends, differences, as well as individual stories, setting
Individuals advocating for and writing about mixed methods research as a distinct, new procedure (e.g., books)

15. They may think that it is analyzing data separately
Quantitative Data
Qualitative Data
Mixing: converging the data or connecting
the data

16. But how do we mix? Converge data: Connect data: Qual Results Quan Qual

17. Why our audience may recognize it
The evidence
Books
Methodological articles
Many published research studies using it
Federal agencies
Private foundations

19. Other writings, initiatives on mixed methods research:
Research studies reported in journals
Methodological articles exploring issues and procedures
Website for bringing mixed methods writers together
Conference sessions
Handbook of Mixed Methods in Social and Behavioral Research
Private foundation interest; federal agency interest

20. NIH Guidelines - Mentioned several approaches for combining
qualitative and
quantitative research
- Considerations for deciding
what model to use (literature
available, prior studies,
realistic design, expertise)
- Need to describe each
method thoroughly

21. Quotes:
“Combining qualitative and quantitative methods has gained broad appeal in public health research. The key question has become not whether it is acceptable or legitimate to combine methods, but rather how they will be combined to be mutually supportive and how findings achieved through different methods will be integrated.” (NIH, Office of Behavioral and Social Science Research, 1999).

22. National Academy of Sciences
Three major research questions in
quality educational research:
What is happening? (qualitative designs)
Is there a systematic effect?
(a quantitative experiment)
Why or how it is happening?
(a qualitative followup)

23. But even if they recognize it, they may not appreciate or understand how to design a mixed methods study
“We are interested in a randomized control trial with a non-experimental approach embedded within it.” (a private foundation officer)
“We accept multi-method studies, but investigators mostly do not sort out the complexity of these projects so that we can understand them.” (a federal projects officer).

24. We need to define mixed methods research for our audiences
Mixed methods research is a design for collecting, analyzing, and mixing both quantitative and qualitative data in a single study or series of studies to understand a research problem.
The purpose of this form of research is that both qualitative and quantitative methods, in combination, provide a better understanding of a research problem or issue than either method alone.

25. Now we could mix within single studies or multiple studies
Single Study:
Quan
Qual
Results
Multiple Studies (called multimethod research):
Quan Qual Qual Quan
Study Study Study Study 4

26. So how do we design a mixed methods study? The model
Worldviews, theoretical frameworks, problem and research
question, skills, resources
Type of mixed methods design
Procedures for:
designing the title
writing the introduction to a study
writing the purpose statement and research
questions/hypotheses
data collection
data analysis
writing the mixed methods report
evaluating the mixed methods research

27. What is a worldview?
Philosophy about your preferences for how you learn about something through research
You prefer the quantitative worldview: you are the expert, you decide what needs to be learned, you build in objectivity
You prefer the qualitative worldview: participant is the expert, participant helps you build knowledge, you bring personal bias in
You prefer both the quantitative and qualitative worldview

28. The next steps in planning our study
Let’s write the overall research question for our study

29. Then let’s choose a type of mixed methods study to conduct
What designs are possible?

30. Types of mixed methods designs
I. Triangulation Mixed Methods Design +
QUAN
Data and Results
QUAL
Data and Results
Interpretation
II. Nested Mixed Methods Design
QUAN
Pre-test
Data and Results
QUAN
Post-test
Data and Results
Qual Process

31. Types of mixed methods designs
III. Explanatory Mixed Methods Design
qual
Data and Results
QUAN
Data and Results
Follow-up
IV. Exploratory Mixed Methods Design
QUAL
Data and Results
quan
Data and Results
Building

32. Triangulation Design: Characteristics
Collecting both quantitative and qualitative data
Collecting these data at the same time in the research procedure
Analyzing the quantitative and qualitative data separately
Comparing or combining the results of the quantitative and qualitative analysis
Example: collect survey data (quantitative) and collect individual interviews (qualitative) and then compare the results

33. Triangulation Design: When is it used?
When you want to combine the advantages of quantitative (trends, large numbers, generalization) with qualitative (detail, small numbers, in-depth)
When you want to validate your quantitative findings with qualitative data
When you want to expand your quantitative findings with some open-ended qualitative data (e.g., survey with closed- and open-ended data)

35. Nested Design: Characteristics
Collecting both quantitative and qualitative data
Collecting both types of data at the same time
Having ONE form of data play a smaller role in the study than the other form of data
Also,
Using one form of data to answer one question; the other form another question
Collecting one form of data at one level of analysis and another at another level of analysis
Example: You conduct an experiment and during the experiment you gather qualitative interview data. The outcomes of the experiment assessed quantitatively address different questions than the process of the experiment explored qualitatively.

36. Nested Design: When is it used?
When you do not have time or resources to commit to extensive quantitative and qualitative data collection
When you want to study the process of an experiment as well as the outcomes
When you want to examine different levels in an organization

37. and analysis of qualitative
Nested Research Design
Experiment
Quan
Data collection
Pre-test
Intervention
Quan
Data collection
Post-test
Process – collection
and analysis of qualitative
data

38. Explanatory Sequential Design: Characteristics
Viewing the study as a two-phase project
Collecting quantitative data first followed by collecting qualitative data second
Typically, a greater emphasis is placed on the quantitative data in the study
Example: You first conduct a survey and then follow up with a few individuals who answered positively to the questions through interviews

39. Explanatory Sequential Design: When do you use it?
When you want to explain the quantitative results in more depth with qualitative data (e.g., statistical differences among groups, individuals who scored at extreme levels)
When you want to identify appropriate participants to study in more depth qualitatively

40. Here is an example of an explanatory design:
Quant
itative
Quantitative
Case Selection
Qualitative
Interpretation –
based on quan
and QUAL results
Data Collection
(quan)
Data Analysis
(quan)
Data Analysis
(QUAL) +
Qualitative
Data Collection
Quantitative Data*
Quantitative Analysis
Case Selection
Qualitative Analysis
Interpretation
Selected 5 cases
maximally varying
Identified critical
months in which
smoking varied
Number of cigarettes
Graphic plot of CES -
Descri
ption of each
Why did changes in
CES - D6
D6 scores over time
case
smoking occur?
for each participant
Identification of life
Qualitative Data*
Graphic plot of
events occurring
Semi -
structured
cigarettes/day values
during critical
over time for each
months where
interviews, audio
recorded and
participant
smoking increased or
decreased
transcribed
Thematic analysis of
life events for each
* Data collected 10 times
case
over the course of a
calendar year for 40
Cross -
case thematic
participants
analysis
Creswell et al. (in progress)

41. Exploratory Sequential Design: Characteristics
Viewing the study as a two-phase project
Qualitative data collection precedes quantitative data collection
Typically, greater emphasis is placed on the qualitative data in the study
Example: You collect qualitative diary entries, analyze the data for themes, and then develop an instrument based on the themes to measure attitudes on a quantitative survey administered to a large sample.

42. Exploratory Sequential Design: When do you use it?
To develop an instrument when one is not available (first explore, then develop instrument)
To develop a classification or typology for testing
To identify the most important variables to study quantitatively when these variable are not known

43. Sequential Exploratory Mixed Methods Design
Phase I Qualitative Research - Year 1
Unstructured Interviews -
50 participants
8 observations at the site
16 documents
Qualitative Data Collection
Qualitative Data Analysis
Text Analysis: Using QSR N6
Development of codes and themes
for each site
Qualitative Findings
Phase II Quantitative Research - Year 2
Create approximately a 80-item
instrument plus demographics
Quantitative Instrument Development
Administer survey to 500 individuals
Determine factor structure of items and
conduct reliability analysis for scales
Quantitative Test of the Instrument
Quantitative Results
Determine how groups differ
using ANOVA test
Sequential Exploratory Mixed Methods Design

44. How will we analyze the quantitative and qualitative data (within the design types)?
Qualitative analysis
Text/image data
Coding
Themes
Description
Interrelated themes
Types of analysis:
Quantitative analysis
Numeric data
Descriptive trend analysis
Hypothesis testing, effect size, interval estimates

45. Triangulation data analysis
QUAN
data collection
QUAL
data collection
Separate QUAN
and QUAL data
analysis
QUAN
data analysis
QUAL
data analysis
Two options
Data transformation (change
QUAL to QUAN or QUAN to QUAL)
Comparison (keep separate and
compare/contrast)
Results

46. Table. Example of Data Transformation of Text Units into Numeric Data
Count
Adj.Count**
Row Pct
Column Pct
Patients
N=2
Physicians
N=4
Medical Assistants
Themes
Familiarity
With the Form 13
6.5
41.9
5.8 17
4.25
27.4
3.0 19
4.75
30.7
5.3 49
15.5
100.0
Reactions to the Form 23
11.5
22.2
10.2
100
25.0
48.3
17.4 61
15.3
29.5
16.9
184
51.75
Use for Managing Depression 67
33.5
38.6
29.8
177
44.25
51.0 36
9.0
10.4
10.0
280
86.75
Changes to the Form
115
57.5
37.5
51.1
196
49.0
32.0
34.0
187
46.8
30.5
51.7
498
153.3
Situational Use of the Form 7
3.5
8.9
3.1 86
21.5
54.4
14.9 58
14.5
36.7
16.1
151
39.5
225
112.5
576
144.0
361
90.3

47. Nested data analysis Quantitative Experiment Intervention Quan
Data collection
Pre-test
Quan
Data collection
Post-test
Qualitative Process
Data
Analysis
Pre-test
scores
Themes/Codes/
Interrelated Themes
Post-test scores
or gain scores
Compare/Describe Results

48. Explanatory sequential data analysis
Qual
data collection
(purposeful sampling)
Select cases based on s.d. variables
Select cases to represent outliers
Select cases to represent extreme cases
Select cases to make group comparisons
QUAN
data analysis
Statistical results
Outlier cases
Extreme cases
Qual
analysis
codes
themes
cases

49. Exploratory sequential data analysis
QUAL data analysis
Quotes
Codes
Themes
Quan data analysis
instrument development
Items on a survey
Variables on a survey
Scales on a survey

50. Let’s identify how we will analyze the data

51. Drawing our Design Let’s draw a picture of our design
Identify the type of design
Add in data collection
Add in data analysis
Show the flow of activities
Add in “products” for our audiences

52. Helpful tips for creating this visual:
Give a title to the visual model.
Choose either horizontal or vertical layout for the model.
Draw boxes for quantitative and qualitative stages of data collection, data analysis and interpretation of the study results.
Use capitalized (QUAN) or small letters (quan) to designate priority of quantitative and qualitative data collection and analysis.
Use single-headed arrows to show the flow of procedures in the design.
Specify procedures for each quantitative and qualitative data collection and analysis stage.
Specify expected products or outcomes of each quantitative and qualitative data collection and analysis procedure.
Make your model simple.
Size your model to one page.

53. Now let’s rework our purpose statement using some scripts

54. Sample Script for a Concurrent Design (Triangulation or Nested)
“The purpose of this concurrent mixed methods study is to better understand a research problem by converging both quantitative (numeric) and qualitative (text or image) data. In this approach, ___________ (quantitative instruments) will be used to measure the relationship between the ________ (independent variables) and __________ (dependent variables). At the same time in the study, the __________ (central phenomenon) will be explored using _____________ (qualitative interviews, documents, observations, visual materials) with _________ (participants) at ____________ (the research site).”

55. Sample Script for a Sequential Exploratory Design
“The purpose of this two-phase, exploratory mixed methods study will be to explore participant views with the intent of using this information to develop and test an instrument with a sample from a population. The first phase will be a qualitative exploration of a _______(central phenomenon) by collecting ___________(data) from ____________ (participants) at _______ (research site). Themes from this qualitative data will then be developed into an instrument (or survey) so that the __________ (theory and research questions/hypotheses) can be tested that ________ (relate, compare) ____________ (independent variable) with __________ (dependent variable) for _________(sample of a population) at _________ (research site).”

56. Sample Script for a Sequential Explanatory Design
“The purpose of this two-phase, explanatory mixed methods study will be to obtain statistical, quantitative results from a sample and then follow-up with a few individuals to probe or explore those results in more depth. In the first phase, quantitative research questions or hypotheses will address the relationship or comparison of __________ (independent) and ________ (dependent) variables with ___________ (participants) at ___________(the research site). In the second phase, qualitative interviews or observations will be used to problem significant _______(quantitative results) by exploring aspects of the ________ (central phenomenon) with
_______ (a few participants) at ____________ (research site).”

57. Criteria for evaluating our plan:
Use appropriate terminology for title and design
Provide a rationale for mixing and include it early in the study (“when you use…”)
Create a mixed methods purpose statement
Identify types of qual and quan data to be collected and qual and quan data analysis steps
Include a visual/procedural diagram of methods with timeline
Use rigorous procedures for the quantitative data collection and analysis

58. Let’s share our drawings of our mixed methods procedures

59. How to Design a Mixed Methods Study
by John W. Creswell, Ph.D. and Vicki L. Plano Clark, M.S.
University of Nebraska-Lincoln
Andrews University, July, 2004