Methodological Issues in Mixed Methods Data

1
Methodological Issues in Mixed Methods Data The
Use of Qualitative and Quantitative Data in
Health Services Research. Susan Zickmund,
PhD Director, Qualitative Research Core CHERP, VA
Pittsburgh Division of General Internal
Medicine University of Pittsburgh susan.zickmund_at_v
a.gov
2
Goal for the Cyber Seminar

• Briefly describe the ABCs of qualitative
  research.
• Provide an introductory guide to mixed methods
  designs.
• Suggest practical ideas on how to best
  transform qualitative themes into numerical
  information and then to integrate that into a
  final analysis plan.

3
Organization of the Seminar

• Traditional qualitative description, data types,
  methods, sample size, recruitment, coding.
• Mixed methods traditional designs, an
  Integrated Mixed Methods approach (with
  methods, sample size, recruitment, coding), and
  conclude with examples of integrated mixed
  statistical models.

4
Traditional Qualitative Approach
5
Description of Qualitative

• The focus is on the participants subjective
  viewpoints.
• Words/images are the primary data elements.
• The approach is inductive.
• Theory development may be a main outcome of the
  analysis.

6
Central Characteristics

• Having an iterative /open ended approach to the
  data.
• Observations or coding schemes emerge directly
  from the text.
• The researcher strives to avoid bias when
  interpreting the data.

7
Classic Components

• Types of data, qualitative methods, determining
  sample size, recruitment, and coding
  philosophies.

8

• Data Types
• Interviews/Focus Groups.
• Observational.

9
Interviews / Focus Groups
Interviews Allow in-depth discussion with one
participant effective for sensitive topics
interviewer controls the discussion. Focus Group
Allows participants to interact group dynamics
provide unique insight moderator has less
control over discussion with one participant.
10
Observational

• A complex situation where a researcher would
  need to observe what is occurring in order to
  best understand the situation.
• Example Hand washing study.
• Participants may have reasons for
  dishonesty.
• The activity is open and observable.

11
Qualitative Methodologies

• Important to have one to guide data collection
  and analysis.
• Types include

• Grounded Theory.
• Ethnography.

12
Grounded Theory

• Most prominent method in medicine.
• Uses constant comparisons between cases.
• Can change recruitment goals based on previous
  findings.
• The goal is an emerging theory.

13
Ethnography

• Method of anthropologists.
• Involves field notes goal to observe and
  understand a culture.

• Effective for an unique or unknown cultural
  dimension of medicine.
• Requires a research question best fitted for
  this method.

14
Sample Size

• Uses thematic saturation idea that once no
  new themes arise, data collection is complete.
• Minimum sample size for saturation is around
  15-20.
• Maximum sample size is any size interfering
  with case oriented thrust of qualitative
  research (60-100).

15
Recruitment Purposeful Sampling

• The goal of recruitment is to purposefully
  section special cases.
• It does not seek generalizability.
• Types of sampling used include
• Extreme / maximum variation.
• Snowball sampling.

16

• Coding Philosophies
• Single investigator.
• Research team approach.
• Independent coders.

17
Mixed Methods Approach
18
Mixed Methods Terminology

• Multiple types of qualitative data or using
  experts with different academic backgrounds
  (triangulation).
• Newer Integrating qualitative and quantitative
  data collection together.

19
The Qualitative-Quantitative Divide

• To some, qualitative is seen as incommensurate
  with empirical data.
• Thus, there is a need to conduct the qualitative
  study in a mixed methods design so as to best
  overcome this divide.

20
Mixed Methods Designs
Time and emphasis (in CAPS).

• qual ? QUANT
• qual preliminary

• quant ? QUAL
• quant preliminary

• QUANT ? qual
• qual follow-up

• QUAL ? quant
• quant follow-up

21
Mixed Methods Designs
A smaller qualitative study designed to provide
data for a larger quantitative one (often survey
based).

• qual ? QUANT
• qual preliminary

• quant ? QUAL
• quant preliminary

• QUANT ? qual
• qual follow-up

• QUAL ? quant
• quant follow-up

22
Mixed Methods Designs
A small quantitative study that is the set- up
for the major qualitative study to follow.

• qual ? QUANT
• qual preliminary

• quant ? QUAL
• quant preliminary

• QUANT ? qual
• qual follow-up

• QUAL ? quant
• quant follow-up

23
Mixed Methods Designs
A major quantitative study that uses qualitative
data to gain insight into its findings.

• qual ? QUANT
• qual preliminary

• quant ? QUAL
• quant preliminary

• QUANT ? qual
• qual follow-up

• QUAL ? quant
• quant follow-up

24
Mixed Methods Designs
A major qualitative study that uses a follow-up
quantitative study at the end.

• qual ? QUANT
• qual preliminary

• quant ? QUAL
• quant preliminary

• QUANT ? qual
• qual follow-up

• QUAL ? quant
• quant follow-up

25
Rarely Integrated at Analytic Level

• These studies are sequential.
• Participants infrequently complete all portions
  of the data.
• These cases do not lend themselves readily to
  analytic integration.

26
Simultaneous Design
Where qualitative and quantitative methods
reinforce simultaneously.

• qual ? QUANT
• qual preliminary

• quant ? QUAL
• quant preliminary

• QUANT ? qual
• qual follow-up

• QUAL ? QUANT
• performed _at_ same time

27
Simultaneous Gold Standard

• Quantitative (demographics, surveys, clinical)
  and qualitative data is collected from all
  participants.
• Analysis plan integrates the quantitative /
  qualitative data together.
• Few examples, but is the best method for fully
  interpreting data in an empirical study.

28
Integrating Mixed Methods (IMM) Overview

• Provide practical approach to
• Research design.
• Analytic strategies that best
• facilitate integrating qualitative
• data into empirical studies.

29
Data Type, Qualitative Methodology

• The data type used would be dictated by the
  research question and may not differ.
• One qualitative method that lends itself well is
  the quasi-statistical method by Crabtree and
  Miller, a methodological approach developed for
  health research.
• Doing Qualitative Research, 1992.

30
Sample Size and Recruitment

• Sample size, rather than using thematic
  saturation, would be determined by sample size
  calculation.
• Recruitment, rather than using purposeful
  sampling, would be consistent with clinical
  research, using inclusion/exclusion criteria.
• Goal is generalizability.

31
Codebook Construction

• Use well-defined methods
• Inclusion/exclusion criteria for codes.
• Sample 20-100 of cases for the codebook
  construction.
• Audit trail, time/date stamping.
• Goal is transparency of method.

32
Coding Philosophy

• Independent coders (two).
• Agreement model to adjudicate differences (need
  final master file).
• Inter-coder kappa statistic to measure
  reliability.
• Goal is reliability.

33
Interpreting Kappa Statistics

• 0.00 poor
• 0.01-0.20 slight
• 0.21-0.40 fair
• 0.41-0.61 moderate
• 0.61-0.80 substantial
• 0.81-1.00 almost perfect
• Rule of thumb shoot for 0.70 and above

34
Data Transformation

• Convert thematic analysis into present / absent
  (0, 1).
• Convert thematic analysis into Likert scales.

35
Computer Data Management

• Software programs (Atlas.ti, Nudist) allow for
  computerized management of
• Interview/focus group files.
• Codebooks.
• Codes.
• Enables a level of textual complexity not
  possible with notes alone.

36
Computer Data Management
37
Computerized Data Output

• Atlas includes the ability to output data to
  tables and spreadsheets (Excel, SPSS).
• Atlas uses 0/1 for presence and absence of codes
  (Likert scales require adaptation).

38
Computerized Data Output

• Useful for master files for integration with
  other data, and facilitates intercoder
  reliability files.

39
IMM Summary

• Simultaneous mixed methods, where the data is
  collected from all participants.
• Sample size calculation over saturation.
• Recruitment generalizability over purposive
  sampling.
• Transparency of codebook/coding methods.
• Intercoder reliability kappa statistics.
• Computerized management spreadsheets.

40
Examples of Using Qualitative Data in Statistical
Models

• Qualitative data as a predictor variable.
• Qualitative data as an outcome variable.

41
Patient Narrative Study
The impact of chronic disease on cancer patients'
self conception
IMM Simultaneous mixed methods design 1 hour
semi-structured interview, survey data,
demographics, clinical data.
42
Qualitative Interview

• View of Self Code
• As you go through this experience, have you
  begun to think about yourself differently?
• Prompts used to guide beyond yes/no.

43
Additional Quantitative Data

• Mortality data (current).
• Charleston Comorbidity data.
• Cancer Staging (time of interview).
• Demographics (self report).
• Sickness Impact Profile (sub-scales).
• Hospital Anxiety Depression Scale
  (Anxiety/Depression scores).

44
Distribution of Answers

• Data available on 825 participants for
• View of Self code
• Better View 22.5
• Unchanged View 49.8
• Worse View 27.6

45
Example of Better View of Self

• I'm back to realizing that I do have an internal
  strength that it will take me wherever I need to
  go in this journey. And it will be a good
  journey, whatever the end outcome is.

46
Example of Worse View of Self

• I am not the person that I was (cries). Just to
  grasp the concept that at a young age you're
  disabled, just like overnight, is very hard to
  swallow. That's a very hard thing to tell
  someone Too bad, your life is ruined, you just
  better learn to go on. And at 37, you're
  thinking Oh my gosh, I just had a baby. 

47
Univariate Results for Predictors of Mortality

• View of Self lt0.001
• Age lt0.001
• Employed 0.001
• gt High School education 0.007
• Cancer staging lt0.001
• Sickness Impact Profile
• Ambulation 0.017
• Appetite 0.004
• Physical Sub-scales 0.011

48
Final Multivariable Model for Predictors of
Mortality

• Variables p-value
• View of self 0.080
• Age lt0.001
• Cancer Stage lt0.001
• SIP-Appetite 0.005
• SIP-Ambulation 0.008

49
Racial Disparities QI Project

• Racial Disparities in Satisfaction with VA Care
• Zickmund SL, Burkitt KH, Rodriguez KL, Switzer
  GE, Stone RA, Shea JA, Gao S, Bayliss N, Meiksin
  R, McClenney LM, Powell CT, Newsome ES, Allen R,
  Fine MJ.
• Center for Health Equity Research and Promotion,
  VA Pittsburgh Healthcare System and Philadelphia
  VA Medical Center VA Center for Minority
  Veterans (CMV) VHA Office of the Assistant
  Deputy Undersecretary for Health

50
Background

• The 2008 VHA Hospital Report Card revealed racial
  disparities in veteran satisfaction with VA
  health care.
• CHERP and the Center for Minority Veterans were
  commissioned to identify reasons for the
  disparity between African Americans and whites.

51
Objectives

• To determine whether racial differences in
  satisfaction existed in overall, outpatient, and
  inpatient VA care.
• To describe racial differences in satisfaction in
  eight domains of health care quality.

52
Design

• Multi-site QI of 30 white/30 African American
  veterans (20 per site).
• Telephone interviews with Likert scale and
  open-ended questions.
• Demographic data collected.

53
Qualitative Methods

• Interviews were coded by 2 coders using an
  iteratively developed codebook.
• Intercoder reliability statistic was
  Kappa0.99.
• Coded themes were then aggregated within the 8
  health care domains, with a distinction between
  satisfied and dissatisfied.

54
8 Health Care Quality Domains

• Trust in provider
• Pain management
• Feelings of respect
• Access to medical care
• Communication with providers
• Coordination of care
• Involvement of family and friends.
• Role of race

55
Domain Access to Care

• One of the things thats a concern for me
  individually right now is that Im trying to get
  a primary care doctor now, and thats like,
  wellI havent had one, and Ive been attending
  the VA off and on for six, seven years.

56
Domain Role of Race

• Veteran A lot of times, they, especially people
  of color and black, Hispanics, Latino, et cetera
  like that, they the providers have a tendency
  to act like were lying, or you want to get high,
  or youre tryingyou know, its almostyouve got
  to either act out or cry or some kind of way to
  validate
• Q That youre really in pain.
• Veteran Yes, yes.

57
Statistical Analysis of Qualitative Data

• 1. Using Chi Square statistics on codes.

58
(No Transcript)
59
Statistical Analysis of Qualitative Data

• Using statistical modeling.
• Item response theory approach (the Rasch
  model).
• Fit random intercept logistic models were used
  to assess the differences between African
  American and white veterans accounting for domain
  and dissatisfaction/satisfaction themes.

60
(No Transcript)
61
Conclusion

• Mixed methods allow the richness of qualitative
  themes to be used along with quantitative data.
• Sequential designs facilitate combining
  qualitative and quantitative work, but do so in a
  segmented way.
• IMM approach enables the integration of the
  qualitative data at the level of the statistical
  analysis.

62
Questions?
Susan Zickmund, PhD Director, Qualitative
Research Core CHERP, VA Pittsburgh susan.zickmund_at_
va.gov 412-954-5259