Introduction to Qualitative Research

1
Introduction to Qualitative Research

• Aaron Louie, User Experience Architect

2

• PURPOSE
• WHY QUALITATIVE RESEARCH?
• RESEARCH DESIGN
• VALIDITY RELIABILITY
• WRITING INTERVIEW QUESTIONS
• ANALYZING DATA
• RESOURCES

3

• PURPOSE
• WHY QUALITATIVE RESEARCH?
• RESEARCH DESIGN
• VALIDITY RELIABILITY
• WRITING INTERVIEW QUESTIONS
• ANALYZING DATA
• RESOURCES

4
What this presentation will do

• Lay the basic groundwork for gaining skills in
  user research
• Provide useful guidelines for writing interview
  questions
• Point you to resources for deeper study in
  qualitative research methods

5
What it wont do

• Substitute for a college course on the subject
• Make you an expert

6

• PURPOSE
• WHY QUALITATIVE RESEARCH?
• RESEARCH DESIGN
• VALIDITY RELIABILITY
• WRITING INTERVIEW QUESTIONS
• ANALYZING DATA
• RESOURCES

7
The purpose of qualitative research is
8
to make sense of reality
9
in a valid and reliable fashion.
10
Describing reality

• Reality is complex.
• All real problems are wicked problems.
• Two ways to approach it
• Quantify it. Then construct mathematical models
  describing it.
• Experience it qualitatively. Then make sense of
  it.

11
Quantitative vs. Qualitative

• Controlled environment
• Experimentation
• Deductive
• Numbers
• Researcher is objective

• Natural/complex environments
• Interaction
• Inductive
• Words
• Researcher is a participant

12
The Scientific Method
Hypothesize
Experiment
Analyze Data
Refine Hypothesis
13
Quantitative Research
Hypothesize
Experiment
Analyze Data
Refine Hypothesis
Predict outcome design experiment to test
outcome
Collect measurements in a controlled environment
Apply statistical calculations
Identify patterns, revise assumptions
14
Qualitative Research
Hypothesize
Experiment
Analyze Data
Refine Hypothesis
Identify assumptions formulate research
questions
Collect data via observation, interviews,
surveys, etc.
Count, classify, structure data
Identify themes, revise assumptions
15

• PURPOSE
• WHY QUALITATIVE RESEARCH?
• RESEARCH DESIGN
• VALIDITY RELIABILITY
• WRITING INTERVIEW QUESTIONS
• ANALYZING DATA
• RESOURCES

16
Research Design

• Determine scope
• What youre researching
• What youre not
• Identify assumptions
• Models from literature
• Researchers biases
• Draft research questions
• Choose your methods
• Data collection methods
• Analysis methods

17
Framing Research Questions

• List what you expect to find out, based on the
  scope and your assumptions.
• For each finding, write a question (or series of
  questions) that must be answered to support or
  invalidate the finding.
• Determine dependencies (which questions must be
  answered first).

18
Qualitative Data Collection Methods

• Surveys
• Pros quantifiable, lower chance of affecting
  subject
• Cons no interaction with subject, nuances of
  emotion or environment lost
• Naturalistic observation
• Pros gain understanding of environment without
  interfering with subject
• Cons very little interaction with subject, no
  insight into subjects mental processes
• Contextual inquiry (shadowing, on-site
  interviews, think-aloud, etc.)
• Pros balanced level of interaction with subject,
  insight into subjects thoughts in context of
  tasks and environment
• Cons researchers physical presence can affect
  subjects behavior
• Ethnography (embedded research)
• Pros maximum level of interaction with subject,
  researcher gains deep understanding by becoming a
  subject, subjects treat researcher as peer
• Cons time- and cost-prohibitive

19
Analysis Methods

• Statistical Analysis
• Content Analysis
• Narrative Analysis
• Strategic Analysis
• Workflow Analysis
• Task Analysis
• Business Analysis
• Cognitive Work Analysis
• Systems Analysis
• Grounded Theory
• etc.

20

• PURPOSE
• WHY QUALITATIVE RESEARCH?
• RESEARCH DESIGN
• VALIDITY RELIABILITY
• WRITING INTERVIEW QUESTIONS
• ANALYZING DATA
• RESOURCES

21
Good research design

• Maximize validity
• Conclusions are true
• Maximize reliability
• Findings are repeatable
• Relatively easy to do in quantitative research
• Impossible to do in qualitative research

22
Threats to validity

• Changes in environment
• Changes in society
• Effects of time
• Changes in person
• Bias in subject
• Interaction between subjects
• (In other words, stuff you cant control.)

23
Threats to validity(stuff you CAN control)

• Bias in researcher
• Interaction between researcher and subject

24
Validity vs. Understanding
25
How does the subject view you?
26

• PURPOSE
• WHY QUALITATIVE RESEARCH?
• RESEARCH DESIGN
• VALIDITY RELIABILITY
• WRITING INTERVIEW QUESTIONS
• ANALYZING DATA
• RESOURCES

27
Composing an Interview Script

• List what must be known in order to answer each
  research question
• For each item in that list, write a question
• Organize the questions into categories
• Sort general ? specific within each category

28
Phrasing Questions

• Ask open questions for general questions
• what, why, when, how, who
• Ask closed questions for specific questions
• how often, how many
• Reserve yes/no questions for branching points in
  interview script
• (i.e., a line of questioning dependent on a
  previous answer)
• Have you ever done X? If so, how did you feel
  about

29
Refining Interview Questions

• Remove biased or charged language
• What do you think about the problems with the
  intranet
• Avoid asking leading questions
• Wouldnt it be better if you did it this way?
• Be aware of your own biases!
• How have consultants like Ascentium helped you
  get your job done better?
• Structure questions to facilitate data analysis

30
Conducting the Interview

• Gain informed consent from the interviewee
• Record audio/video whenever possible
• Remain open-minded dont think too much
• Listen, listen, listen
• Avoid reacting too much (negatively or
  positively) to the interviewees responses
• Take prodigious notes
• Write up notes immediately after interview

31

• PURPOSE
• WHY QUALITATIVE RESEARCH?
• RESEARCH DESIGN
• VALIDITY RELIABILITY
• WRITING INTERVIEW QUESTIONS
• ANALYZING DATA
• RESOURCES

32
Analyzing The Data

• Go through notes transcript
• Identify frequent keywords
• Categorize responses
• Look for patterns
• Match keywords, categories, and patterns to
  models and assumptions

33

• PURPOSE
• WHY QUALITATIVE RESEARCH?
• RESEARCH DESIGN
• VALIDITY RELIABILITY
• WRITING INTERVIEW QUESTIONS
• ANALYZING DATA
• RESOURCES

34
Resources

• Books
• The Craft of Research by Wayne C. Booth, Joseph
  M. Williams, Gregory G. Colomb link
• Qualitative Research Methods for the Social
  Sciences by Bruce L. Berg link
• Cognitive Work Analysis Toward Safe, Productive,
  and Healthy Computer-Based Work by Kim J. Vicente
  link
• Web sites
• Cognitive Work Analysis Toward Safe, Productive,
  and Healthy Computer-Based Work by Kim J. Vicente
  link
• Qualitative Research Consultants Association
  FAQs about Qualitative Research link
• Association for Information Systems Qualitative
  Resarch in Information Systems link
• UsabilityNet Tools and methods link
• UW Libraries Lit Reviews, Research Methods
  link
• Virginia Tech Annotated Bibliography on
  Qualitative Research Methodology link