Reliability and Validity in Qualitative Research aka Credibilty and Trustworthiness

1
Reliability and Validity in Qualitative Research
(aka Credibilty and Trustworthiness)

• Ellen Olshansky, DNSc, RNC, FAAN
• Professor Chair, Dept. of Health Community
  Systems
• University of Pittsburgh School of Nursing

2
Objectives of Presentation

• To briefly describe the naturalistic-interpretive
  paradigm as the foundation for conducting and
  critiquing qualitative research
• To describe specific criteria used to evaluate
  qualitative research studies based on this
  paradigm

3
Naturalistic-Interpretive Paradigm

• Attempt to generate theory that will reflect and
  explain the data collected (as opposed to
  verifying existing theory)
• Philosophic belief in multiple perspectives,
  multiple realities there is not one truth
• Recognition that bias exists in interpretation
  of data
• Concept of pragmatism or usefulness

4
Qualitative Concepts of Validity and Reliability

• Credibility ensuring that the theoretical
  framework generated is understood and is based on
  the data from the study
• Usefulness worthwhile end product to help
  explain a phenomenon
• Trustworthiness extent to which one can believe
  in the research findings
• Glaser and Strauss, 1967

5
Qualitative Concepts of Validity and Reliability

• Truth value adequate representation of multiple
  constructions of reality (recognizing that these
  are reconstructions based on researcher)
• Applicability fittingness or transferability
• Consistency dependability ability of other
  researchers to follow the method used
• Neutrality auditability
• Lincoln Guba (1985)
• Sandelowski (1986)

6
Qualitative Concepts of Validity and Reliability

• Comparability when the researcher clearly
  describes characteristics of group studied as
  well as the concepts generated, enabling
  comparisons to be made with other groups.
• Translatability ability to make comparisons
  confidently by clearly explaining the research
  methods, analytic categories, characteristics of
  group studied.
• LeCompte Goetz (1982)

7
Strategies for Achieving Trustworthiness/Credibili
ty in Qualitative Research

• Prolonged engagement with and observation of
  informants
• Triangulation (multiple sources of data)
• Peer debriefing (colleagues)
• Negative case analysis (to include commonalities
  as well as variabilities)
• Referential adequacy (theoretical sampling)
• Member checks (research participants/informants)
• Employing an auditor
• Thick description (to reflect complexities in the
  data)
• Prevention of premature foreclosure on the data
• Maintaining a journal to enhance self-reflection

8
Prolonged engagement with and observation of
informants

• Need to develop a trusting relationship with
  research participants
• Need to observe and interact in various contexts
  over time
• Need to get a deep and complex understanding of
  the phenomenon under study

9
Triangulation

• Multiple methods of data collection (interviews
  individual and group, observation, literature,
  archives)
• Multiple investigators
• Multiple contexts/situations

10
Peer Debriefing

• Share data with colleagues (those who are experts
  in the field of study and those who are not)
• May ask peers to code a few transcripts
• May ask peers to listen to the analysis you are
  in the process of developing ask for feedback

11
Negative Case Analysis

• There are not outliers in qualitative research
• Embrace all the variabilities
• Learn from the negative cases what explains
  why this case, this person is different from the
  others? leads to a more complex, dense, thick
  analysis

12
Referential Adequacy

• Theoretical sampling sample various groups
  based on the ongoing data reference groups
• This is a process that occurs as the research
  progresses certain referent groups are
  described and studied

13
Member Checks

• Going back to the informants to see if the
  analysis/interpretation makes sense to them,
  reflects their experiences
• May go back to the actual participants or to
  other informants who you have not previously
  interviewed, or both

14
Employing an Auditor

• An outside person who can verify the steps you
  went through in arriving at your data
  analysis/interpretation
• Verify the logic of your chronology of the
  research process able to outline the steps
• Verify that a systematic process was undertaken

15
Thick Description

• The analysis/interpretation of data should be
  thick in that it includes the complexities
  the variabilities as well as the commonalities
• The analysis represents the diversity of
  perspectives among the research participants,
  leading to an interpretation that includes these
  variabilities under varying contexts/conditions

16
Prevention of Premature Closure on the Data

• Continue data collection and analysis until
  theoretical saturation is reached
• Provide evidence of theoretical saturation
• Generate questions for further study indicating
  what areas have not been answered yet

17
Maintaining a Journal to Enhance Self-Reflection

• Keep track of your own ideas, responses, biases
  in order to try as best as you can to separate
  your responses from the responses of the
  participants
• Acknowledge your own biases, locate yourself in
  the data
• Continue to be self-reflective though not a
  naval-gazer!

18
Criteria for Evaluating Trustworthiness/Credibilit
y in Qualitative Research

• Evidence of knowledge on the part of the
  researcher appropriateness of conducting a
  qualitative study (what gaps are being
  addressed)?
• Acknowledgment of experiential data
• Evidence of incorporating several comparable
  groups (systematic, theoretical sampling,
  systematic narrowing of focus)

19
Criteria for Evaluating Trustworthiness/Credibilit
y in Qualitative Research

• Evidence of systematically formulating
  provisional hypotheses and the data to support
  them (and interview questions becoming more
  focused)
• Evidence of having reached theoretical
  saturation and data to support this saturation
  (do not prematurely foreclose on the data)
• Empirical data must be presented throughout the
  presentation of results

20
Criteria for Evaluating Trustworthiness/Credibilit
y in Qualitative Research

• Phenomenon recognition achieved through
  providing evidence of having gone back to the
  field
• Presentation of theoretical model generated
  should be clear (perhaps a figure that represents
  the model)

21
Summary of Major Points

• Evaluating and critiquing qualitative research
  (establishing validity and reliability) are
  based on the paradigm from which qualitative
  methods have been developed
• Need to evaluate validity and reliability in
  terms of the concepts of trustworthiness and
  credibility
• Specific techniques for enhancing trustworthiness
  and credibility
• Specific criteria upon which to evaluate
  trustworthiness and credibility