Evaluating Qualitative Research

1
Evaluating Qualitative Research

• INFO 272. Qualitative Research Methods
• 16 April 2009

2
Typical Reactions

• is not generalizable / is anecdotal
• The sample is too small to say anything / is not
  a random sample / not representative
• What is the hypothesis you are testing?
• Great stories, but can you show me some data that
  supports your claims?
• is subjective, the researchers presence in the
  setting biases the data
• lacks rigor, procedure is unsystematic

3
Becker epistemology of qual research
Quantitative Tradition Qualitative Tradition
Reliability reproducing the findings through the same procedures, same findings from multiple observers Accuracy based on close observation not remote indicators
Validity the degree to which one measured the phenomenon one claims to be dealing with Precision close to the thing discussed
Breadth knowledge of a broad range of matters that touch on the topic
4
Criteria for Quant Research
5
Functional Equivalence

• Criteria for evaluating quantitative research is
  not directly applicable to qualitative research
• Can we draw out some abstract, general standards
  and then respecify for qualitative research
• Kvale on epistemology
• Abandoning a correspondence theory of truth
• Defensible (rather than absolute) knowledge
  claims requiring argumentation

6
Functional Equivalence
Quantitative Tradition Qualitative Tradition
Reliability of measures (c) Confidence (c) Relevance (r) Triangulation and reflexivity (c)
Internal validity (c) Confidence (c) Relevance (r) Transparency and procedural clarity (c)
Sample size (c) Confidence (c) Relevance (r) Corpus construction (c, r)
Representative sampling (r) Confidence (c) Relevance (r) Thick description (c, r)
External validity (r) Confidence (c) Relevance (r) Local surprise (r)
Validity of measures (r) Confidence (c) Relevance (r) Communicative validation (r)
7
Triangulation and Reflexivity (c)

• In situ verification process
• i.e. interviews about Internet use supplemented
  by observation

8
Transparency (c)
9
Corpus Construction (c, r)

• Maximizing the diversity of unknown
  representations and mapping those representations
• Representativeness and external validity is a
  matter of argumentation

10
Thick Description (c, r)

• high-fidelity reportage verbatim quotes
  demonstrating the provenance of a claim
• Footnotes and sources
• But also, do you get a whole picture of the
  social world, its elements, and how they are
  interlinked? Especially the meaning of the
  social phenomenon.

11
Local Surprise (r)

• Surprise in relation to a common-sense view
• Surprise in relation to theoretical expectation
• Solely confirming evidence (just as totally
  consistent evidence) should raise suspicion

12
Communicative Validation (r)

• Gaining feedback from research participants (and
  others?)
• Remember interviewing technique of interpreting
  on the fly to get confirmation from interviewees

13
The Future of Evaluation

• Websites and digital archives that make
  qualitative data accessible to the public

14
Summary

• Make your methods visible
• Make your data (ideally) available
• Continual verification in situ (as part of your
  iterative process)
• Closeness to the social phenomenon and openness
  to surprises, the counter-intuitive
• Re-read Becker on the epistemology of
  qualitative research for further suggestions