Methodology and Explanation XX50125

1
Methodology and ExplanationXX50125

• Lecture 2
• Part 1 Usability testing
• Part 2 Interviews and questionnaires
• Dr. Danaë Stanton Fraser

2
Lectures

1. Introduction to Methods
2. Experiments
3. Usability testing
4. Interviews/questionnaires
5. Ethnography

3

• Know thy user Hansen (1971).

4
What is Usability Testing?

• Usability testing is a systematic approach to
  evaluating user performance in order to inform
  design.
• Designs should be
• Based on observation of users
• Refined by thoughtful analysis of the frequency
  and sequences in which tasks are carried out
• Validated through early prototype, usability
  tests
• Direct interaction with users during the design
  phase and throughout the system lifecycle

5
The Move towards Usability Testing

• Traditionally, computer programmers designed for
  computer programmers
• Designs should be
• Based on observation of users
• Refined by thoughtful analysis of the frequency
  and sequences in which tasks are carried out
• Validated through early prototype, usability
  tests
• Direct interaction with users during the design
  phase and throughout the system lifecycle

6

• Iterative design allowing for early testing of
  prototypes, revisions based on feedback and then
  incremental refinements
• When competitive products provide similar
  functionality usability testing important for
  product acceptance
• Karat (1994) at IBM reported up to 100 payoffs
  for each dollar spent on usability, with benefits
  in reduced program-development costs, reduced
  programme maintenance costs, increased revenue
  due to higher customer satisfaction, and improved
  user efficiency and productivity.

7

• Design is a process..

8
Example Designing Games

• Interdisciplinary team audio and visual design,
  programming, project management etc
• Iterative design process
• cost benefits
• does design process prevent creative coding?
• Design process should be a shared vision that is
  built upon

9
ExampleDesigning Games

• A videogame takes approx. 30 to 36 months from
  inception to market. A Pre-production phase
  (prototyping) will generally take 12 months plus,
  and if it meets approval, will get funded for
  full development
• A PS2 videogame in the US will typically have a
  team of 50 plus people working on it, and will
  cost 5m plus (Jeffery, 2003, Lucas Arts)

10
Usability Testing and Laboratories

• Emerged since the early 1980s
• To provide expert reviews
• To conduct usability tests of products
• Usability testing often found to increase speed
  of projects and produce cost savings
• Usability tests are designed to find flaws in
  user interfaces
• Includes set of tasks, fewer participants than
  experimental study and a report of recommended
  changes.

11
Typical procedure

• Usability staff
• Participate in early task analysis/design review
• Provide information on software tools, literature
• Develop set of tasks for testing (numbers, types
  of participants)
• Pilot test
• Voluntary participation with informed consent
• Think aloud technique
• Video recording

12
Many types of Usability Testing

• Discount usability testing
• Field tests
• Paper prototypes
• Can-you-break-this
• Competitive usability testing
• Two main limitations
• Emphasises first time use
• Limited coverage of features
• Therefore often supplemented with expert review

13
Expert Reviews

• Heuristic evaluation
• Guidelines review
• Consistency inspection
• Cognitive walkthrough
• Formal usability inspection

14
Part 2 Interviews and questionnaires
15
Why Interview?

• Good for obtaining information about
• Opinions
• Feelings
• Goals (organisational, personal and system)
• Things you didnt know you needed to know
• Bad for gathering (large volumes of) data

16
Interview Preparation

• Decide objectives
• Read background material
• Decide whom to interview
• Set up the interview
• Decide interview structure
• Write down questions you will ask and pointers
  for questions you might ask

17
Open Questions

• E.g. How do you find using this technology in
  your lectures?
• Little control
• Hard to analyse
• Hard to keep focussed on interview objectives
• May unsettle interviewee
• Time consuming and can be difficult to analyse
• However
• May gain more depth, rich data
• May tell you things you had not considered
• May give you ideas for future projects

18
Closed Questions

• E.g. How many courses do you teach?
• How many times have you used this technology?
• May settle interviewee
• Easy to analyse
• Easy to maintain focus
• Observation and use of data may be more efficient
• However
• Data may not be as rich

19
Question Pitfalls

• Leading questions
• How bad is the current system?
• Double questions/poor structure
• How often does the system fail, and how do you
  fix it when it does?
• Stay focussed. If you are unlikely to be able to
  make use of the answer dont ask the question!

20
Question Structure

• Pyramid
• Start with closed questions, finish with open
• Good for settling nervous interviewees as answers
  are easy early on
• Funnel
• Start with open questions, finish with closed
• Good for aggressive interviewees as opinions are
  asked for early on
• Diamond
• Closed -gt Open -gt Closed, Good all-round
  technique
• Structured/unstructured
• All interviews may vary depending on how focused
  you intend to be on the prepared structure

21
Starting the Interview

• Dress
• Greeting first time
• Who you are
• Why you want to interview them
• What objectives you want to achieve from the
  interview
• That the information they give is confidential
  and how it might be used
• Subsequent greetings should discuss an issue from
  or summarise the previous interview
• Dont just be yourself

22
During the Interview

• Introduce each line of questioning
• Probing
• E.g. Why?
• Shows that you are listening
• Good for keeping interview focus
• Double-check information and notes
• Taking notes
• Not too many
• Maintain eye contact
• Go back over important points
• Use of sound recorders

23
Focus Groups

• Normally 3-10 people
• A representative sample of typical users
• Enables people to put forward ideas in a
  supportive environment
• Facilitator guides and prompts
• Discussion recorded
• Low-cost but need good facilitator

24
Questionnaires

• For wider coverage
• Need to know the right questions (through
  interviews and background)
• Focus on objectives
• Dont ask unnecessary questions
• Keep it short and facilitate response
• Open and Closed questions
• Scaling

25

• Likert Scale
• Short statements about the features to be
  evaluated
• Divide the items into groups with same number of
  positive and negative statements in each group
• Decide on scale even or odd, number of points

26
Questionnaire difficulties

• The halo effect
• Respondent answers everything the same by not
  differentiating between questions
• Mix questions up on different subjects
• Leniency
• Some respondents rate everything too well
• Move the average category left (or right) of
  centre and make scale descriptors stronger
• Central tendency
• Respondents rate everything as average
• Increase the number of points on the scale and
  adjust the scale descriptors to cover a smaller
  range of opinions

27
Alsoevaluation during active use

• System refinement as a result of experience or in
  response to changes in users
• interviews and focus group discussions
• continuous user-performance data logging
• look at both frequent and infrequent error
  messages
• analyse sequences of actions to suggest
  improvements or new actions
• BUT respect peoples rights and consult them
  first!
• User feedback mechanisms
• on-line forms, email and bulletin boards
• workshops and conferences

28
Preparation for Practical