Interviews and Questionnaires

1
Interviews and Questionnaires

• a.k.a. How to talk to your users

2
Agenda

• Questions
• Project Part 3 review
• Overview of techniques (and reminder from earlier
  lectures)
• Interviewing techniques
• Questionnaire design
• Evaluating your data

3
Questions

• IRB
• Midterms
• P2
• P3

4
P3

• Evaluation Plan
• Use a lot of techniques
• Interviews and questionnaires
• Lab tests with think aloud, software logging,
  etc.
• Prototypes
• Recommend at least three
• Can be low-fi drawings, storyboards, scenarios
• Or hi-fi Flash, VB, Java
• Or physical foam, playdough, duct tape ?
• BUILD PROTOTYPES THAT ALLOW YOU TO TEST WHAT YOU
  NEED TO TEST AND DO YOUR EVALUATION RIGHT

5
Review Why evaluation is done?

• Formative
• assess a system being designed
• gather input to inform design
• Summative
• assess an existing system
• judge if it meets some criteria

• Summative or formative?
• Depends on
• maturity of system
• how evaluation results will be used
• Same technique can be used for either

6
Making an evaluation plan

• What criteria are important?
• What resources available?
• evaluators, prototype, subjects
• Required authenticity of system

7
Evaluation techniques

• Predictive modeling
• Questionnaire
• Empirical user studies
• Heuristic evaluation
• Cognitive walkthrough
• Think aloud (protocol analysis)
• Interviews

8
Todays focus is asking people about stuff

• Interviews
• Questionnaires
• Assessment of that data

9
Guidelines for both

• Consider the ordering of the questions
• Avoid complex/long/multiple questions
• Avoid jargon talk in participants language

10
Interview design

• Summative or formative
• Quantitative or qualitative
• Usually inexpensive way to get detailed
  information

11
Preparing for your interviews

• Get buy-in/entrée from your participants
• Develop a rapport beforehand if possible
• Dress similarly to your participants
• Meet on their ground (if possible)
• Have your informed consent ready and available
• Test your recording equipment

12
A good interview follows a plan

• Introduction
• Warmup
• Main session
• Cool-off
• Closing
• Record everything exactly in your participants
  languages

13
Types of interviews

• Structured
• Efficient
• Require training specific to interview needs
• Unstructured
• Inefficient
• No new specific training (outside of the training
  that one needs to be a good unstructured
  interviewer)
• Semi-structured
• Good balance
• Often appropriate
• Group
• Focus groups

14
Ethnographic Interviews

• Have a question plan, but keep interview open to
  different directions
• Be specific
• Create interpretations with users
• Be sure to use their terminology
• At end, query What should I have asked?
• Record interviews

15
Structured Interviews

• More similar to questionnaires
• Require a lot of training for any hope at
  inter-interviewer reliability
• But that means that they tend to give much more
  repeatable results

16
Semi-Structured Interviews

• Predetermine data of interest - know why you are
  asking questions - dont waste time
• Plan for effective question types
• How do you perform task x?
• Why do you perform task x?
• Under what conditions do you perform task x?
• What do you do before you perform?
• What information do you need to?
• Whom do you need to communicate with to ?
• What do you use to?
• What happens after you?
• See Gordon Gill, 1992 Graesser, Lang,
  Elofson, 1987

17
Whats wrong with this picture?

• How much easier is it to use this email client
  than Outlook?
• I see you choose to use your keyboard shortcuts
  more than the mouse. Is that faster for you?
• Your choice of red is different than any other
  user we saw. Why did you do that?

18
Questionnaire design

• Summative or formative
• Quantitative or
• qualitative
• Usually inexpensive
• way to get lots of
• information (not
• necessarily detailed)

Remember
19
However,

• Questionnaire design is difficult
• Lets talk about the surveys some of you did in
  P1

20
Contents of a survey

• General/Background info
• name, experience
• Objective
• Open-ended/subjective

21
Review you can have a lot of different forms of
responses

• Dichotomous
• Multiple Choice
• Multiple Response
• Rank/Match
• Likert
• Semantic Differential
• Rating

22
Advice on survey design

• Take your own survey first
• Know what answers you are trying to elicit
• Too long, and youll be sorry

23
Some more advice..

• Make your questions clear and specific you
  cant follow up at the time of the inquiry
• Include a none/NA/no opinion option
• When using a range, make sure it is appropriate
  and non-overlapping
• Provide clear instructions on completion
• Balance white space and compacting questionnaire

24
Whats wrong with this picture?

• 2. State your age in years_____
• 3. How long have you used the internet?
• lt1 year
• 1-3 years
• 3-5 years
• gt5 years
• 4. Do you use the Web to
• Purchase goods
• Send email
• Visit chatrooms
• Use bulletin boards
• Find information
• Read the news
• 5. How useful is the internet to you?
• __________________________________________________
  _________
• __________________________________________________
  _________

25
On line questionnaires

• Email or internet
• Change checkboxes into dropdowns, etc
• Take advantage of the technology check input
• Ensure as accessible as paper (browser and email
  client compatibility)
• Ensure confidentiality how is this different
  from paper?

26
Using interviews and questionnaires later in the
design process

• With a (semi-) working protoype
• I felt I could accomplish all of the tasks I was
  given easily
• I would be willing to use this software/device/kio
  sk/etc to perform these types of tasks in the
  future
• What are some tasks for which you could imagine
  using this software/device/kiosk/etc?

27
Using interviews and questionnaires later in the
design process

• With non-working protoypes (scenarios,
  storyboards, foam, playdough, etc)
• I could imagine using a device of this size (form
  factor, weight, etc) in those situations
• Describe how you would carry this device in your
  daily life.
• I would feel comfortable using this software in
  my home.

28
Lets practice

• Divide into pairs.

29
Analyzing your data

• Transcribe your interviews
• code the open-ended responses
• Ensure an appropriate level of inter-rater
  reliability
• Enter your data into something you can manipulate
• Excel
• SPSS
• Look for trends in the data
• Make charts, etc
• Run statistical analysis
• Use lo-fi mnethods (post-its, affinity diagrams,
  etc)