How to Design a Mixed Methods Study

1
How to Design a Mixed Methods Study

• by John W. Creswell, Ph.D. and Vicki L. Plano
  Clark, M.S.
• University of Nebraska-Lincoln
• Andrews University, July, 2004

2
(No Transcript)
3
How would you combine two types of data?

• Quantitative Numeric Data
• 2342543112232132
• 23322543
• 3122432432132433
• 32334441
• 2222111432143213
• 22111555
• 2331432432132433
• 32135432

• Qualitative Text Data
• This is a sample of a text file of words that
  might be collected on transcripts through
  interviews, fieldnotes from observations, or from
  optically-scanned documents.

4
Objectives of the workshop

• Lets design a mixed methods study
• Lets study how people learn mixed methods
  research in this room? (or you can work on your
  own project and follow along at each step)
• Lets start with a title. Write a title.
• What data will we collect?

5
What are types of quantitative and qualitative
data?

• Quantitative data
• Close-ended scales
• Attitudinal/behavioral scales
• Behavioral checklists
• Census, attendance records

• Qualitative data
• Open-ended responses
• Semi-structured interviews
• Semi-structured observations
• Records/documents
• Videotapes

6

• Lets identify our quantitative and qualitative
  data collection

7
Now lets consider some reasons for why we are
collecting (and mixing) both forms of data

• Together quantitative and qualitative data
  provide both precise measurement and
  generalizability of quantitative research and the
  in-depth, complex picture of qualitative research
• To validate quantitative results with qualitative
  data
• We do not have an adequate instrument. Thus, we
  need to explore views and develop an instrument
• Our quantitative data provide a general
  explanation and we need to follow-up with
  participants and have them explain the
  quantitative results
• In our experiment, outcomes to be measured are
  not enough they need to be complemented by
  understanding the process of participants

8

• Lets identify our reason for mixing

9
So

• There are good reasons for gathering both forms
  of data
• Butthere are certain requirements for this to
  work best

10
Requirement 1 Now lets consider whether we
have the skills, time, and resources?

• We need minimum skills in both qualitative and
  quantitative data collection. What do we need?
• We need time and resources for extensive data
  collection and analysis. How much time and
  resources do we need?

11

• Write down the skills, time, and resources we
  will need

12
Requirement 2 The audience(s)

• Does our audience appreciate both numbers and
  stories?
• Are they familiar with this design?
• Do they need to be educated?
• Are examples of published studies available in
  our content area?

13

• Lets identify the audiences

14
But audiences may not recognize it yet because it
is so new

• Increased use and acceptance of qualitative
  research from 1990s to present
• The complexity of our research problems today
  requires understanding trends, differences, as
  well as individual stories, setting
• Individuals advocating for and writing about
  mixed methods research as a distinct, new
  procedure (e.g., books)

15
They may think that it is analyzing data
separately
Quantitative Data
Qualitative Data
Mixing converging the data or connecting the
data
16
But how do we mix?

• Converge data

Qual
Quan
Results
Connect data
Qual
Quan
Results
17
Why our audience may recognize it

• The evidence
• Books
• Methodological articles
• Many published research studies using it
• Federal agencies
• Private foundations

18
(No Transcript)
19
Other writings, initiatives on mixed methods
research

• Research studies reported in journals
• Methodological articles exploring issues and
  procedures
• Website for bringing mixed methods writers
  together
• Conference sessions
• Handbook of Mixed Methods in Social and
  Behavioral Research
• Private foundation interest federal agency
  interest

20
NIH Guidelines
- Mentioned several approaches for
combining qualitative and quantitative
research - Considerations for deciding what
model to use (literature available, prior
studies, realistic design, expertise) - Need to
describe each method thoroughly
21
Quotes

• Combining qualitative and quantitative methods
  has gained broad appeal in public health
  research. The key question has become not whether
  it is acceptable or legitimate to combine
  methods, but rather how they will be combined to
  be mutually supportive and how findings achieved
  through different methods will be integrated.
  (NIH, Office of Behavioral and Social Science
  Research, 1999).

22
National Academy of Sciences

• Three major research questions in
• quality educational research
• What is happening? (qualitative designs)
• Is there a systematic effect?
• (a quantitative experiment)
• Why or how it is happening?
• (a qualitative followup)

23
But even if they recognize it, they may not
appreciate or understand how to design a mixed
methods study

• We are interested in a randomized control trial
  with a non-experimental approach embedded within
  it. (a private foundation officer)
• We accept multi-method studies, but
  investigators mostly do not sort out the
  complexity of these projects so that we can
  understand them. (a federal projects officer).

24
We need to define mixed methods research for our
audiences

• Mixed methods research is a design for
  collecting, analyzing, and mixing both
  quantitative and qualitative data in a single
  study or series of studies to understand a
  research problem.
• The purpose of this form of research is that both
  qualitative and quantitative methods, in
  combination, provide a better understanding of a
  research problem or issue than either method
  alone.

25
Now we could mix within single studies or
multiple studies

• Single Study

Qual
Quan
Results
Multiple Studies (called multimethod research)
Quan Qual Qual
Quan
Study 1 Study 2 Study 3
Study 4
26
So how do we design a mixed methods study? The
model
Worldviews, theoretical frameworks, problem and
research question, skills, resources

• Type of mixed methods design
• Procedures for
• designing the title
• writing the introduction to a study
• writing the purpose statement and research
• questions/hypotheses
• data collection
• data analysis
• writing the mixed methods report
• evaluating the mixed methods research

27
What is a worldview?

• Philosophy about your preferences for how you
  learn about something through research
• You prefer the quantitative worldview you are
  the expert, you decide what needs to be learned,
  you build in objectivity
• You prefer the qualitative worldview
  participant is the expert, participant helps you
  build knowledge, you bring personal bias in
• You prefer both the quantitative and qualitative
  worldview

28
The next steps in planning our study

• Lets write the overall research question for our
  study

29
Then lets choose a type of mixed methods study
to conduct

• What designs are possible?

30
Types of mixed methods designs
I. Triangulation Mixed Methods Design

QUAN Data and Results
QUAL Data and Results
Interpretation
II. Nested Mixed Methods Design
QUAN Post-test Data and Results
QUAN Pre-test Data and Results
Qual Process
31
Types of mixed methods designs
III. Explanatory Mixed Methods Design
qual Data and Results
QUAN Data and Results
Follow-up
IV. Exploratory Mixed Methods Design
QUAL Data and Results
quan Data and Results
Building
32
Triangulation Design Characteristics

• Collecting both quantitative and qualitative data
• Collecting these data at the same time in the
  research procedure
• Analyzing the quantitative and qualitative data
  separately
• Comparing or combining the results of the
  quantitative and qualitative analysis
• Example collect survey data (quantitative) and
  collect individual interviews (qualitative) and
  then compare the results

33
Triangulation Design When is it used?

• When you want to combine the advantages of
  quantitative (trends, large numbers,
  generalization) with qualitative (detail, small
  numbers, in-depth)
• When you want to validate your quantitative
  findings with qualitative data
• When you want to expand your quantitative
  findings with some open-ended qualitative data
  (e.g., survey with closed- and open-ended data)

34
(No Transcript)
35
Nested Design Characteristics

• Collecting both quantitative and qualitative data
• Collecting both types of data at the same time
• Having ONE form of data play a smaller role in
  the study than the other form of data
• Also,
• Using one form of data to answer one question
  the other form another question
• Collecting one form of data at one level of
  analysis and another at another level of analysis
• Example You conduct an experiment and during
  the experiment you gather qualitative interview
  data. The outcomes of the experiment assessed
  quantitatively address different questions than
  the process of the experiment explored
  qualitatively.

36
Nested Design When is it used?

• When you do not have time or resources to commit
  to extensive quantitative and qualitative data
  collection
• When you want to study the process of an
  experiment as well as the outcomes
• When you want to examine different levels in an
  organization

37
Nested Research Design
Experiment
Intervention
Quan Data collection Post-test
Quan Data collection Pre-test
Process collection and analysis of
qualitative data
38
Explanatory Sequential Design Characteristics

• Viewing the study as a two-phase project
• Collecting quantitative data first followed by
  collecting qualitative data second
• Typically, a greater emphasis is placed on the
  quantitative data in the study
• Example You first conduct a survey and then
  follow up with a few individuals who answered
  positively to the questions through interviews

39
Explanatory Sequential Design When do you use
it?

• When you want to explain the quantitative results
  in more depth with qualitative data (e.g.,
  statistical differences among groups, individuals
  who scored at extreme levels)
• When you want to identify appropriate
  participants to study in more depth qualitatively

40

Here is an example of an explanatory design




Quant
itative
Interpretation based on quan and QUAL results

Qualitative
Case Selection

Quantitative

Data Collection (quan)

Data Analysis (QUAL)

Data Analysis (quan)







Qualitative

Data Collection





Quantitative Data

Quantitative Analysis

Case Selection

Qualitative Analysis

Interpretation

Selected 5 cases maximally varying Identified
critical months in which smoking varied
Number of cigarettes

Graphic plot of CES
-
Descri
ption of each





Why did changes in
D6 scores over time

case

smoking occur?


CES
-
D6

for each participant





Identification of life

Qualitative Data


events occurring
Graphic plot of

during critical
cigarettes/day values

Semi
-
structured

months where
over time for each
interviews, audio
smoking increased or


participant
recorded and

decreased


transcribed


Thematic analysis of


life events for each


Data collected 10 times
case


over the course of a


Cross
-
case thematic
calendar year for 40


analysis

participants





Creswell et al. (in progress)

41
Exploratory Sequential Design Characteristics

• Viewing the study as a two-phase project
• Qualitative data collection precedes quantitative
  data collection
• Typically, greater emphasis is placed on the
  qualitative data in the study
• Example You collect qualitative diary entries,
  analyze the data for themes, and then develop an
  instrument based on the themes to measure
  attitudes on a quantitative survey administered
  to a large sample.

42
Exploratory Sequential Design When do you use it?

• To develop an instrument when one is not
  available (first explore, then develop
  instrument)
• To develop a classification or typology for
  testing
• To identify the most important variables to study
  quantitatively when these variable are not known

43

Phase I Qualitative Research - Year 1
Unstructured Interviews - 50 participants 8
observations at the site 16 documents
Qualitative Data Collection
Qualitative Data Analysis
Text Analysis Using QSR N6
Development of codes and themes for each site
Qualitative Findings
Phase II Quantitative Research - Year 2
Create approximately a 80-item instrument plus
demographics
Quantitative Instrument Development
Administer survey to 500 individuals Determine
factor structure of items and conduct
reliability analysis for scales
Quantitative Test of the Instrument
Quantitative Results
Determine how groups differ using ANOVA test
Sequential Exploratory Mixed Methods Design
44
How will we analyze the quantitative and
qualitative data (within the design types)?

• Qualitative analysis
• Text/image data
• Coding
• Themes
• Description
• Interrelated themes

• Types of analysis
• Quantitative analysis
• Numeric data
• Descriptive trend analysis
• Hypothesis testing, effect size, interval
  estimates

45
Triangulation data analysis
QUAN data collection
QUAL data collection
QUAN data analysis
QUAL data analysis

• Separate QUAN
• and QUAL data
• analysis

• Two options
• Data transformation (change
• QUAL to QUAN or QUAN to QUAL)
• Comparison (keep separate and
• compare/contrast)

Results
46
Table. Example of Data Transformation of Text
Units into Numeric Data
Count Adj.Count Row Pct Column Pct Patients N2 Physicians N4 Medical Assistants N4
Themes
Familiarity With the Form 13 6.5 41.9 5.8 17 4.25 27.4 3.0 19 4.75 30.7 5.3 49 15.5 100.0
Reactions to the Form 23 11.5 22.2 10.2 100 25.0 48.3 17.4 61 15.3 29.5 16.9 184 51.75 100.0
Use for Managing Depression 67 33.5 38.6 29.8 177 44.25 51.0 30.7 36 9.0 10.4 10.0 280 86.75 100.0
Changes to the Form 115 57.5 37.5 51.1 196 49.0 32.0 34.0 187 46.8 30.5 51.7 498 153.3 100.0
Situational Use of the Form 7 3.5 8.9 3.1 86 21.5 54.4 14.9 58 14.5 36.7 16.1 151 39.5 100.0
225 112.5 100.0 576 144.0 100.0 361 90.3 100.0
47
Nested data analysis
Quantitative Experiment
Intervention
Quan Data collection Post-test
Quan Data collection Pre-test
Qualitative Process
Data Analysis
Pre-test scores
Post-test scores or gain scores
Themes/Codes/ Interrelated Themes
Compare/Describe Results
48
Explanatory sequential data analysis

• Qual
• data collection
• (purposeful sampling)
• Select cases based on s.d. variables
• Select cases to represent outliers
• Select cases to represent extreme cases
• Select cases to make group comparisons

• QUAN
• data analysis
• Statistical results
• Outlier cases
• Extreme cases

• Qual
• analysis
• codes
• themes
• cases

49
Exploratory sequential data analysis
QUAL data analysis Quotes Codes Themes
Quan data analysis instrument development Items
on a survey Variables on a survey Scales on a
survey
50

• Lets identify how we will analyze the data

51
Drawing our Design

• Lets draw a picture of our design
• Identify the type of design
• Add in data collection
• Add in data analysis
• Show the flow of activities
• Add in products for our audiences

52
Helpful tips for creating this visual

1. Give a title to the visual model.
2. Choose either horizontal or vertical layout for
   the model.
3. Draw boxes for quantitative and qualitative
   stages of data collection, data analysis and
   interpretation of the study results.
4. Use capitalized (QUAN) or small letters (quan) to
   designate priority of quantitative and
   qualitative data collection and analysis.
5. Use single-headed arrows to show the flow of
   procedures in the design.
6. Specify procedures for each quantitative and
   qualitative data collection and analysis stage.
7. Specify expected products or outcomes of each
   quantitative and qualitative data collection and
   analysis procedure.
8. Make your model simple.
9. Size your model to one page.

53
Now lets rework our purpose statement using some
scripts

54
Sample Script for a Concurrent Design (Triangulati
on or Nested)
The purpose of this concurrent mixed methods
study is to better understand a research problem
by converging both quantitative (numeric) and
qualitative (text or image) data. In this
approach, ___________ (quantitative instruments)
will be used to measure the relationship between
the ________ (independent variables) and
__________ (dependent variables). At the same
time in the study, the __________ (central
phenomenon) will be explored using _____________
(qualitative interviews, documents, observations,
visual materials) with _________ (participants)
at ____________ (the research site).
55
Sample Script for a Sequential Exploratory Design
The purpose of this two-phase, exploratory mixed
methods study will be to explore participant
views with the intent of using this information
to develop and test an instrument with a sample
from a population. The first phase will be a
qualitative exploration of a _______(central
phenomenon) by collecting ___________(data) from
____________ (participants) at _______ (research
site). Themes from this qualitative data will
then be developed into an instrument (or survey)
so that the __________ (theory and research
questions/hypotheses) can be tested that ________
(relate, compare) ____________ (independent
variable) with __________ (dependent variable)
for _________(sample of a population) at
_________ (research site).
56
Sample Script for a Sequential Explanatory Design
The purpose of this two-phase, explanatory mixed
methods study will be to obtain statistical,
quantitative results from a sample and then
follow-up with a few individuals to probe or
explore those results in more depth. In the
first phase, quantitative research questions or
hypotheses will address the relationship or
comparison of __________ (independent) and
________ (dependent) variables with ___________
(participants) at ___________(the research site).
In the second phase, qualitative interviews or
observations will be used to problem significant
_______(quantitative results) by exploring
aspects of the ________ (central phenomenon)
with _______ (a few participants) at ____________
(research site).
57
Criteria for evaluating our plan

• Use appropriate terminology for title and design
• Provide a rationale for mixing and include it
  early in the study (when you use)
• Create a mixed methods purpose statement
• Identify types of qual and quan data to be
  collected and qual and quan data analysis steps

• Include a visual/procedural diagram of methods
  with timeline
• Use rigorous procedures for the quantitative data
  collection and analysis

58
Lets share our drawings of our mixed methods
procedures

59
How to Design a Mixed Methods Study

• by John W. Creswell, Ph.D. and Vicki L. Plano
  Clark, M.S.
• University of Nebraska-Lincoln
• Andrews University, July, 2004