The Validity and Reliability of Qualitative Research

1
The Validity and Reliability of Qualitative
Research
2
Training Overview

• Section 1 Philosophical Orientation Getting in
  the Right Mindset
• Section 2 Enhancing the Validity of Qualitative
  Research
• Section 3 Enhancing the Reliability of
  Qualitative Research
• Section 4 Applications of Qualitative Research
  Design
• Section 5 Group Work to Apply Learning

3
Section 1

• Philosophical Orientation

• The philosophical underpinnings of qualitative
  research (as contrasted against quantitative
  research)
• Implications for sampling
• Implications for addressing issues of validity
  and reliability

4
Quantitative and Qualitative Paradigms Contrasted

• Quantitative Research is
• Fundamentally an inferential enterprise that
  seeks to uncover universal principles
• Philosophically and methodologically built or
  designed around the ability to infer from a
  sample to a larger population

5
Quantitative and Qualitative Paradigms Contrasted

• Qualitative Research is
• Fundamentally an interpretive enterprise that is
  context-dependent
• Philosophically and methodologically built or
  designed around the ability to interpret
  (comprehend/understand) a phenomenon from an emic
  (insider), as well as an etic (outsider)
  perspective
• This inter-subjective (i.e., shared)
  understanding serves as a proxy for objectivity

6
Implications for Research Questions

• A client wants to know about how well a program
  is working for youth
• Quantitative Research Questions (descriptive,
  explanatory -gtinferential)
• Qualitative Research Questions
  (descriptive, explanatory -gtinterpretive)

7
Implications for Sampling

• Sampling Strategies used in Quantitative
  Research
• Obtaining a random or representative sample
  (based on probabilities)
• Permits the researcher to infer from a segment of
  the population (from which it is more feasible to
  collect data) to a larger population

8
Implications for Sampling

• Sampling Strategies used in Qualitative Research
• Purposive sampling (to ensure that the researcher
  has adequately understood the variation in the
  phenomena of interest)
• Theoretical sampling (to test developing ideas
  about a setting by selecting phenomena that are
  crucial to the validity of those ideas)
• Example Case study selection in ACY (see, also,
  handout on sampling techniques)

9
Implications for Handling Threats to Validity and
Reliability

• In quantitative research, threats to validity are
  addressed by prior design features (such as
  randomization and controls)

10
Implications for Handling Threats to Validity and
Reliability

• In qualitative research, such prior elimination
  of threats to validity is less possible because
• qualitative research is more inductive, and
• it focuses primarily on understanding particulars
  rather than generalizing to universals.
• Qualitative researchers view threats as an
  opportunity for learning
• - e.g. researcher effects and bias are part of
  the story that is told they are not controlled
  for

11
Section 2

• Implications
• Enhancing the Validity
• of Qualitative Research

• Defining validity within the qualitative paradigm
• Major types of validity within the qualitative
  paradigm
• Design considerations

12
Validity

• Validity is not a commodity that can be purchased
  with techniques Rather, validity is, like
  integrity, character and quality, to be assessed
  relative to purposes and circumstances.

Brinberg and McGrath 198513
13
Validity

• In general, validity concerns the degree to which
  an account is accurate or truthful
• In qualitative research, validity concerns the
  degree to which a finding is judged to have been
  interpreted in a correct way

14
Assessing the Validity of Qualitative Research

• Can another research read your field (and other
  types of) notes (i.e., the explication of your
  logic) and come to the same understandings of a
  given phenomenon?
• Concern about validity (as well as reliability)
  is the primary reason thick description is an
  essential component of the qualitative research
  enterprise
• Resources
• Handout Different Types of Notes
• Example ACY Site Visit Toolkit

15
Major Types of Validity in Qualitative Research

• Descriptive Validity
• Interpretive Validity
• Theoretical Validity
• External Validity (i.e., generalizability)

16
Descriptive Validity

• Concerned with the factual accuracy of an account
  (that is, making sure one is not making up or
  distorting the things one hears and sees)
• All subsequent types of validity are dependent on
  the existence of this fundamental aspect of
  validity

17
Descriptive Validity

• Behavior must be attended to, and with some
  exactness, because it is through the flow of
  behavior or, more precisely, social action
  that cultural forms find articulation.

Geertz 197317
18
Interpretive Validity

• Interpretive accounts are grounded in the
  language of the people studied and rely, as much
  as possible, on their own words and concepts
• At issue, then, is the accuracy of the concepts
  as applied to the perspective of the individuals
  included in the account

19
Interpretive Validity Design Consideration

• While the relevant consensus about the terms used
  in description rests in the research community,
  the relevant consensus for the terms used in
  interpretation rests, to a substantial extent, in
  the community studied
• An important design element, for increasing
  interpretive validity, therefore, is to employee,
  at some level/to some degree, a participatory
  research approach (e.g., through member checks,
  peer to peer research model, etc.)

20
Theoretical Validity

• Theoretical understanding goes beyond concrete
  description and interpretation its value is
  derived based on its ability to explain
  succinctly the most amount of data
• A theory articulates/formulates a model of
  relationships as they are postulated to exist
  between salient variables or concepts
• Theoretical validity is thus concerned, not only
  with the validity of the concepts, but also their
  postulated relationships to one another, and thus
  its goodness of fit as an explanation

21
Major Threats to Validity

• Type I error believing a principle to be true
  when it is not (i.e., mistakenly rejecting the
  null hypothesis)
• Type II error rejecting a principle when, in
  fact, it is true
• Type III error asking the wrong question

22
Triangulation An Important Theoretical Validity
Check

• Case example Parable of the blind men and the
  elephant

23
Triangulation An Important Theoretical Validity
Check

• The most fertile search for validity comes from a
  combined series of different measures, each with
  its own idiosyncratic weaknesses, each pointed to
  a single hypothesis. When a hypothesis can
  survive the confrontation of a series of
  complementary methods of testing, it contains a
  degree of validity unattainable by one tested
  within the more constricted framework of a single
  method.

Webb et al. 1966174
24
External Validity in Qualitative Research

• There is broad agreement that generalizability
  (in the sense of producing laws that apply
  universally) is not a useful standard or goal for
  qualitative research
• This is not to say, however, that studies
  conducted to examine a particular phenomenon in a
  unique setting cannot contribute to the
  development of a body of knowledge accumulating
  about that particular phenomenon of interest
• Consensus appears to be emerging that for
  qualitative researchers generalizability is best
  thought of as a matter of the fit between the
  situation studied and others to which one might
  be interested in applying the concepts and
  conclusions of that study.

25
Enhancing External Validity

• Thick descriptions are crucial.
• Such descriptions of both the site in which the
  studies are conducted and of the site to which
  one wishes to generalize (or apply ones
  findings) are critical in allowing one to search
  for the similarities and differences between the
  situations.
• Analysis of these similarities and differences
  makes it possible to make a reasoned judgment
  about the extent to which we can use the findings
  from one study as a working hypothesis about what
  might occur in another situation.

26
Multi-site Studies Another Way to Enhance
Generalizability

• A finding emerging repeatedly in the study of
  numerous sites would appear to be more likely to
  be a good working hypothesis about some as yet
  unstudied site than a finding emerging from just
  one or two sites.
• A finding emerging from the study of several very
  heterogeneous sites would be more robust and,
  thus, more likely to be useful in understanding
  various other sites than one emerging from the
  study of several very similar sites.
• Heterogeneity may be obtained by creating a
  sampling frame that maximizes the variation
  inherent in the sample, specifically in terms of
  potentially theoretically important dimensions

27
Section 3

• Implications
• Enhancing the Reliability
• of Qualitative Research

• Defining reliability
• Key strategies for enhancing the reliability of
  qualitative research

28
Reliability

• Reliability concerns the ability of different
  researchers to make the same observations of a
  given phenomenon if and when the observation is
  conducted using the same method(s) and
  procedure(s)

29
Enhancing the Reliability of Qualitative Research

• Researchers can enhance the reliability of their
  qualitative research by
• Standardizing data collection techniques and
  protocols
• Again, documenting, documenting, documenting
  (e.g., time day and place observations made)
• Inter-rater reliability (a consideration during
  the analysis phase of the research process)

30
Section 4

• Applications of
• Qualitative Research Design

• Core Qualitative Methods
• Guiding Principles
• Qualitative Research Techniques

31
Core Qualitative Methods

• Semi- or Un-structured, Open-Ended
• In-depth Interviews
  (in the field, face-to-face)
• Participant Observation
  (field/site visits)
• Archival Research
  (document review and analysis)

32
Guiding Principles

• Qualitative research designs consider ways to
  foster
• Reflexivity (an ongoing process of reflecting on
  the researchers subjective experience, ways to
  broaden and enhance this source of knowing,
  examining how it informs research)
• Iteration (a spiraling process sequential and
  repetitive steps in examining preliminary
  findings for the purposes of guiding additional
  data collection and analysis)
• Intersubjectivity (a process of reaching a
  shared/ objective agreement about how to assign
  meaning to a social experience - with insiders
  and outsiders)

33
The Iterative Process of Qualitative Research A
Model
Analysis
Data Collection
Reflection
34
Qualitative Research Techniques

• Instrumentation
• Key Informants (question development and piloting
  of instrument)
• Unstructured to Semi-structured
• Probing
• Data Processing and Analytic Tools

35
Qualitative Research Techniques/ Considerations

• Sampling
• Single v. Multiple Cases (not an individual)
• Expert and Key Informants (identification and
  recruitment of sample)
• Roles of the Researcher (identification and
  recruitment of sample)

36
Qualitative Research Techniques

• Data Collection
• Participants as Data Collectors
• Field Notes (personal reflections, observations,
  emerging concepts/theories)
• Debriefing (a participant, a participating
  researcher, a non-participating researcher)

37
Qualitative Research Techniques

• Analysis
• Key Informant Feedback
• Codebooks (specifies definitions and
  relationships of concepts and terms)
• Memos (emerging patterns, concepts documentation
  of analytic pathways)
• Case Analysis Meeting (a meeting of a research
  team for the purposes of reflecting on analytic
  process, tools, and findings)
• Matrices or Diagrams (to identify and examine
  time sequencing, the structure of relationships,
  conditions of cross case events)

38
Section 5

• Group Work to Apply Learning

39

• What is a given program achieving with homeless
  and runaway youth?
• Key Methodological Issues
  (instrumentation, sampling, data
  collection, analysis)
• What More Do You Need to Know?
• Initial Methodological Approach and Justification

40
In-House Qualitative Resources

• Handouts, OMNI Reports and Proposals
• Qualitative Research Design, An Interactive
  Approach (Maxwell, 1996) Sage, Applied Social
  Research Methods Series
• Qualitative Data Analysis (Miles Huberman,
  1994)
• The Quality of Qualitative Research (Seale, 1999)
• Focus Groups, Theory and Practice (Stewart
  Shamdasani, 1990) Sage, Applied Social Research
  Methods Series
• The Mismeasure of Man (Gould)