Human Computer Interaction

1
Human Computer Interaction

• Introducing evaluation

2
The aims

• Discuss how developers cope with real-world
  constraints
• Explain the concepts and terms used to discuss
  evaluation
• Examine how different techniques are used at
  different stages of development

3
Two main types of evaluation

• Formative evaluation is done at different stages
  of development to check that the product meets
  users needs
• Summative evaluation assesses the quality of a
  finished product
• Our focus is on formative evaluation

4
What to evaluate

• Iterative design evaluation is a continuous
  process that examines
• Early ideas for conceptual model
• Early prototypes of the new system
• Later, more complete prototypes
• Designers need to check that they understand
  users requirements

5
Bruce Tognazzini tells you why you need to
evaluate

• Iterative design, with its repeating cycle of
  design and testing, is the only validated
  methodology in existence that will consistently
  produce successful results. If you dont have
  user-testing as an integral part of your design
  process you are going to throw buckets of money
  down the drain.
• See AskTog.com for topical discussion about
  design and evaluation

6
When to evaluate

• Throughout design
• From the first descriptions, sketches etc. of
  users needs through to the final product
• Design proceeds through iterative cycles of
  design-test-redesign
• Evaluation is a key ingredient for a successful
  design

7
Approaches Naturalistic

• Naturalistic
• describes an ongoing process as it evolves over
  time
• observation occurs in realistic setting
• ecologically valid
• real life
• External validity
• degree to which research results applies to real
  situations

8
Approaches Experimental

• Experimental
• study relations by manipulating one or more
  independent variables
• experimenter controls all environmental factors
• observe effect on one or more dependent variables
• Internal validity
• confidence that we have in our explanation of
  experimental results
• Trade-off Natural vs Experimental
• precision and direct control over experimental
  design versus
• desire for maximum generalizability in real life
  situations

9
Approaches Reliability Concerns

• Would the same results be achieved if the test
  were repeated?
• Problem individual differences
• best user 10x faster than slowest
• best 25 of users 2x faster than slowest 25
• Partial Solution
• reasonable number and range of users tested
• statistics provide confidence intervals of test
  results
• 95 confident that mean time to perform task X is
  4.5/-0.2 minutes means95 chance true mean is
  between 4.3 and 4.7, 5 chance its outside that

10
Approaches Validity Concerns

• Does the test measure something of relevance to
  usability of real products in real use outside
  of lab?
• Some typical reliability problems of testing vs
  real use
• non-typical users tested
• tasks are not typical tasks
• physical environment different
• quiet lab vs very noisy open offices vs
  interruptions
• social influences different
• motivation towards experimenter vs motivation
  towards boss
• Partial Solution
• use real users
• tasks from task-centered system design
• environment similar to real situation

11
Qualitative Evaluation Techniques
12
Qualitative methods for usability evaluation

• Qualitative
• produces a description, usually in non-numeric
  terms
• may be subjective
• Methods
• Introspection
• Extracting the conceptual model
• Direct observation
• simple observation
• think-aloud
• constructive interaction
• Query via interviews and questionnaires
• Continuous evaluation via user feedback and field
  studies

13
Querying Users via Interviews

• Excellent for pursuing specific issues
• vary questions to suit the context
• probe more deeply on interesting issues as they
  arise
• good for exploratory studies via open-ended
  questioning
• often leads to specific constructive suggestions
• Problems
• accounts are subjective
• time consuming
• evaluator can easily bias the interview
• prone to rationalization of events/thoughts by
  user
• users reconstruction may be wrong

14
Evaluating the 1984 OMS

• Early tests of printed scenarios user guides
• Early simulations of telephone keypad
• An Olympian joined team to provide feedback
• Interviews demos with Olympians outside US
• Overseas interface tests with friends and family.
• Free coffee and donut tests
• Usability tests with 100 participants.
• A try to destroy it test
• Pre-Olympic field-test at an international event
• Reliability of the system with heavy traffic

15
Development of HutchWorld

• Many informal meetings with patients, carers
  medical staff early in design
• Early prototype was informally tested on site
• Designers learned a lot e.g.
• language of designers users was different
• asynchronous communication was also needed
• Redesigned to produce the portal version

16
Usability testing

• User tasks investigated
• how users identify was represented
• communication
• information searching
• entertainment
• User satisfaction questionnaire
• Triangulation to get different perspectives

17
Findings from the usability test

• The back button didnt always work
• Users didnt pay attention to navigation buttons
• Users expected all objects in the 3-D view to be
  clickable
• Users did not realize that there could be others
  in the 3-D world with whom to chat
• Users tried to chat to the participant list

18
Key points

• Evaluation design are closely integrated in
  user-centered design
• Some of the same techniques are used in
  evaluation requirements but they are used
  differently (e.g., interviews questionnaires)
• Triangulation involves using a combination of
  techniques to gain different perspectives
• Dealing with constraints is an important skill
  for evaluators to develop