Data Collection and Analysis

1
Data Collection and Analysis
KMUTT Human Centre Design

• Nantida Wisawayodhin
• nantida24_at_gmail.com
• 4 December 2006
• 930 1230

2
AGENDA
Data Collection and Analysis

• 930 1030 Presentation
• 1030 1045 Break
• 1045 1200 Practical Session
• 1200 1215 Break
• 1215 1230 Wrap Up

3
Aim and Objective
Aim To learn widely used subjective and
objective assessment and analysis techniques in
usability testing in order to obtain meaningful
data to assess the effectiveness, efficiency and
satisfaction of a product

• Objectives
• Gain an understanding of the importance of users
  opinions
• Identify relevant factors and disciplines in
  usability assessment
• Learn how to develop, administer and analyse key
  subjective and objective assessment techniques
• An overview of key factors to consider when
  designing an experiment

4
Overview of Todays Presentation

• Introduction
• Relevant factors and disciplines in usability
  testing
• Subjective data collection methods
• Questionnaires
• Interviews
• Checklists

5
Overview of Todays Presentation
continue

• 4. Objective data collection methods
• Time on task
• Error type and frequency
• 5. Nature of data collected
• 6. Data collation analysis
• Data logger

6
1. INTRODUCTION
The 5 Fundamental Fallacies

• This design is satisfactory for me it will,
  therefore, be satisfactory for everybody else.
• This design is satisfactory for the average
  person it will, therefore, be satisfactory for
  everybody else.
• The variability of human beings is so great that
  it cannot possibly be catered for in any design
  but since people are wonderfully adaptable it
  doesnt matter anyway.
• Usability is expensive and since products are
  actually purchased on appearance and styling,
  usability considerations may conveniently be
  ignored.
• Usability is an excellent idea. I always design
  things with usability in mind but I do it
  intuitively and rely on my common sense so I
  dont need tables of data or empirical studies.

Expert judgements are important, but users
opinions are essential.
7
Usability testing method data collection
1. INTRODUCTION
Usability testing Method
Empirical studies
Expert reviews Heuristic Evaluation
Methods Cognitive Walkthroughs Pluralistic
Walkthroughs Empirical method Formal design
analysis
User participation Performance measures Protocol
analysis
Subjective assessments Questionnaires Interviews C
hecklists Objective assessments Time on
task Error type and frequency
Datalogger Range, mean, median,
mode Percentage Distribution Sample t-test
Bar graphs Line graphs Distribution graphs Pie
chart Tabular form Diagram
8
2. RELEVANT FACTORS AND DISCIPLINES
Disciplines
Factors
9
2. RELEVANT FACTORS AND DISCIPLINES
Continue
Disciplines
Factors
10
3. SUBJECTIVE DATA COLLECTION METHODS
3.1 Questionnaires

• A series of questions concerning usability issues
  designed to be answered in a predetermined order

• There are 2 main types
• Fixed response questionnaires
• Alternative responses
• Rating scales

• Open-ended response questionnaires

When to use When investigator knows specifically
what he/she wants to know or clarify
11
3. SUBJECTIVE DATA COLLECTION METHODS
3.1 Questionnaires

• Content design
• Factual statements
• simple yes/no responses
• Multiple category questions
• categories are specified and user chooses which
  of the categories apply
• Rating scale
• use to assess attitudes of users to specified
  product attributes. Commonly used to assess
  comfort, satisfsction, ease of use, perceived
  degrees of difficulty etc.

12
3. SUBJECTIVE DATA COLLECTION METHODS
3.1 Questionnaires

• Key considerations for the design of
  questionnaire statements
• Validity Does it measure what its supposed to
  measure?
• Reliability Will you get the same answer for
  the same question if administered to the same
  person on two separate occasions?
• Assumptions Education level, culture, language,
  knowledge, experience
• Language Terminology, abstract concept
• Ambiguity and clarity Different people must
  understand the statements the same way. One
  statement should only convey one message.
• Leading questions Do you agree that?, Is it
  easy?, How difficult is it to?

13
3. SUBJECTIVE DATA COLLECTION METHODS
3.1 Questionnaires

• Fixed response questionnaires
• Alternative responses (a set of predefined
  categories)
• Provide participants with choice of answers
• Participants can only tick a box
• Answer choices must cover a full range of
  possible answers

Example How often do you use a product?
Never Rarely Occasionally
Quite often Very often All the time
14
3. SUBJECTIVE DATA COLLECTION METHODS
Continue
3.1 Questionnaires

• Fixed response questionnaires
• Simple rating scales
• Likerts Summated Ratings Method

15
3. SUBJECTIVE DATA COLLECTION METHODS
Continue
3.1 Questionnaires

• Fixed response questionnaires
• Simple rating scales
• Clear and unambiguous statements of attributes to
  be tested. One for each attribute
• Neutral or no
• Clear and unambiguous words for the anchors, or
  leave it if in doubt
• Use 100 mm line
• Watch out for leniency and halo effects
• Data obtained can be quantified
• Most commonly used technique by ergonomists and
  usability specialists

Example How easy was it to read the
instructions? Very easy
Very difficult
16
3. SUBJECTIVE DATA COLLECTION METHODS
Continue
3.1 Questionnaires

• Fixed response questionnaires
• Likerts Summated Ratings Method
• Clearly define a hypothesis e.g. Usability is
  less important than design
• Clear and unambiguous statements about the
  hypothesis being tested e.g. People tend to
  choose beautifully designed products over easy to
  use products
• 10 20 differently phrased statements assessing
  the same attribute is recommended e.g. Easy to
  use products tend to be less attractive than
  beautifully designed products

Example People tend to choose beautifully
designed products over easy to use products
Very easy
Very difficult
17
3. SUBJECTIVE DATA COLLECTION METHODS
Continue
3.1 Questionnaires

• Fixed response questionnaires
• Likerts Summated Ratings Method
• A 5-point scale is used for each statement each
  with a scale value
• Ensure that the points allocated is matched to
  the statement
• The sum of the scale values selected represents
  the users opinion
• Care should be taken in the design of each
  statement
• Requires more experience and effort to generate
  than that for simple rating scale

Example Easy to use products tend to be less
attractive than beautifully designed products
Very easy
Very difficult
18
3. SUBJECTIVE DATA COLLECTION METHODS
Questionnaires

• Open-response questionnaires
• Often used to obtain free responses from users
  without pre-defining the answer
• Wide variety of responses maybe obtained
• Post-hoc coding is often used during analysis
• Can be time consuming to analyse

19
3. SUBJECTIVE DATA COLLECTION METHODS
3.1 Questionnaires

• Pre-prepared questionnaires
• ASQ (After-scenario Questionnaire) (J. Lewis,
  1991b)
• IID (Index of Interactive Difficulty) (Jordan and
  ODonnell, 1992)
• NASA TLX (Task Load Index by North American Space
  Agency) (Hart and Staveland, 1988)
• SUS (System Usability Scale) (Brooke, 1996)
• SUMI (Software Usability Measurement Inventory)
  (Kirakowski, 1996)

20
SUS
21
3. SUBJECTIVE DATA COLLECTION METHODS
3.1 Questionnaires

• Advantages
• Relatively cheap, easy and quick to use
• Obtain large amount of information from a large
  number of people
• Can be used at any stage of a design process
• Anonymity affords honest and to the point answers
• Can be free of investigation/participant effect

• Disadvantages
• Small number repeated and returned (25)
• Unrepresentative sample extreme opinions
• Designing a questionnaire is highly demanding on
  resource and skills
• Risk of ambiguity in the questions abstract and
  technical terms
• Require vigorous validity and reliability testing
• Risk of misinterpretation of the
  questions/statements

22
3. SUBJECTIVE DATA COLLECTION METHODS
3.2 Interviews

• A series of questions developed to be
  administered by the investigator

• There are 3 main types
• Unstructured
• Semi-structured
• Structured

Development and content design is very much the
same with the development of questionnaires
23
3. SUBJECTIVE DATA COLLECTION METHODS
3.2 Interviews

• Unstructured interviews
• Open-ended questions
• Questions are general and broad to give
  participants room to consider a wider application
  of the product being tested
• Participants steer the discussion
• Explorative method suitable for early design
  stages such as requirement capture
• Can be used to identify important usability
  issues or functions of a product

Example What do you like and dislike about this
product? Why do you like/dislike this feature
of the product?
24
3. SUBJECTIVE DATA COLLECTION METHODS
3.2 Interviews

• Semi-structured interview
• Open-ended questions with prompts or specific
  questions
• List of issues or features of particular interest
  can be used as prompts to ensure that these are
  covered
• Investigator intervention when topics or issues
  of interest is being discussed or not being
  discussed
• Suitable for early prototyping stages where
  requirements have been identified, but some key
  points or issues need particular attention

Example For assessment of text, the investigator
may have a list of the various attributes e.g.
font type and size, colour, clarity, ambiguity,
language use etc.
25
3. SUBJECTIVE DATA COLLECTION METHODS
3.2 Interview

• Structured interview
• Close-ended questions
• Yes/no respond or choose a respond from a preset
  range design options, a set of categories,
  rating scale
• Suitable for all stages of a design life cycle
  where investigator has a fairly clear idea of the
  issues that need investigation and is quite clear
  in the answers required
• Similar to fixed response questionnaire

Example Which one of these would describe you?
Novice Occasional user Regular user Expert
26
3. SUBJECTIVE DATA COLLECTION METHODS
3.2 Interview

• Advantages
• Can be used throughout the design process
• Low risk of misinterpretation as participants can
  ask investigator for clarity
• More likely to obtain useful data
• Obtain more valid data
• Can explore unexpected data
• Relatively less preparation effort
• High completion and return rate

• Disadvantages
• High administration cost, especially if a large
  number is required
• Data obtained maybe distorted by an
  investigation/participant effect
• Require an experienced interviewer with knowledge
  about topic area and must be sensitive to people
• Interviewers bias may creep in
• Difficulty in finding and scheduling people
• Data assimilation could take much longer than to
  obtain it

27
3. SUBJECTIVE DATA COLLECTION METHODS
3.3 Feature Checklists
Comprehensive list of product functions or
features to collect data on usability of product
design

• Primarily gives information about the way a
  product is used rather than how easy it is to
  use. But can be designed to incorporate types of
  information that can give an indication of
  usability.
• For example, if a feature is present, but user
  does not know how to use it, then there might be
  a usability problem.

Suitable for requirements capture when developing
product
Another type of checklist - an observational
checklist is designed to be used by the
investigator to collect data during
trial Suitable for user trials with interactive
prototypes
28
3. SUBJECTIVE DATA COLLECTION METHODS
3.3 Checklists

• Content design
• Can be simply a list of product functionality or
  extended to cover
• regularity of use,
• whether users realise how to use a particular
  function
• Whether users know that a particular function
  existed
• Use semantic descriptions or command names to
  describe the tasks
• Visual checklist mirroring the product in some
  useful way could help increasing the validity of
  users response
• Use clear and unambiguous wordings

29
3. SUBJECTIVE DATA COLLECTION METHODS
3.3 Checklist

• Advantages
• Cheap to use
• Relatively easy to create and administer
• An effective means to gain an overview of the way
  in which a product is used
• No laboratory or video equipment necessary
• Undemanding of investigators time

• Disadvantages
• No measurable or quantifiable data
• Simple and not in-depth data
• No information about users experience or opinion
  of the product being tested

30
4. OBJECTIVE DATA COLLECTION METHODS
3.4 Time on task

• The more quickly a user can complete a task with
  a product, the more efficient the product

• One of the most widely used measures of usability
• One of the most meaningful way to measure
  efficiency
• This type of data is usually obtained in
  conjunction with data on error
• Can be used in field observation or in a
  controlled experiment
• Video recording is normally used to obtain
  accurate timeline
• Can be difficult to clearly identify the start
  and the end of some tasks, especially cognitive
  tasks

31
4. OBJECTIVE DATA COLLECTION METHODS
3.5 Error type and frequency
The less the number of errors, the less effort
required which means that the user can complete
the task efficiently.
Error type can be classified by the reasons of
occurrence or by the severity of consequence

• Reasons of occurrence
• - Slips and lapses
• - Mistakes
• Severity of consequence
• - Minor
• - Major
• - Fatal
• - Catastrophic

32
4. OBJECTIVE DATA COLLECTION METHODS
3.5 Error type and frequency
Error type Reasons

• Slips and lapses
• Occurs when the user knows how to perform a task,
  but accidentally does something wrong during the
  task
• Often occur because the interface to the product
  is inadequately designed e.g. a button is placed
  in an inappropriate location
• Example 1 Slip - The users intention is to
  rename the file, but accidentally press the
  delete command from the pull down menu
• Example 2 Lapses The user has an intention to
  zoom in on the image on the camera before taking
  the shot, but forgot to overtly carry out the
  task

33
4. OBJECTIVE DATA COLLECTION METHODS
3.5 Error type and frequency
Error type Reasons

• Mistakes
• Occurs when the user has an erroneous model of
  how a product works
• Occur because the underlying principles of how a
  product works are not intuitive or they are
  counter-intuitive
• Example An elderly lady wants to turn on a video
  player, but instead of pressing the power button,
  she presses the red light indicating the status
  of the video player, believing that this is the
  right control

34
4. OBJECTIVE DATA COLLECTION METHODS
3.5 Error type and frequency
Error type Reasons

• Observable error actions

35
4. OBJECTIVE DATA COLLECTION METHODS
3.5 Error type and frequency
Error type Severity

• Minor
• Errors that the user can notice and correct
  either instantly or within a fairly short period
  of time with no other consequences
• Example The user is intended to pause a video
  tape, but instead of pressing pause, he/she
  presses the forward button. The user notices
  this immediately and rectify in no time
• Major
• Errors that the user is able to spot and
  rectifies with no other consequences, but at a
  greater cost in terms of time and annoyance
• Example The user has to reset the recording
  timer on a VCR because he/she noticed that a
  wrong channel is set.

36
4. OBJECTIVE DATA COLLECTION METHODS
3.5 Error type and frequency
Error type Severity

• Fatal
• Errors that prevents the user from completing the
  task they were attempting, because the action
  prevented the user from recovery or carrying on
• Example The user intending to return back to the
  front page of a mobile phone after having checked
  a message, but pressed the delete button by
  mistake resulting in the message being
  permanently deleted
• Catastrophic
• Errors that not only prevent the task from being
  carried out, but also cause other problems.
• Example The user, after having highlighted a
  word document, pressed enter by mistake and
  consequently deleted the entire content

37
4. SAMPLE OF A DATA COLLECTION FORM
3.2 Interview Example
Time taken s
38
5. Nature of data collected in usability trials
KPI
Data type
Raw data
39
6.1 Data Logging
6. DATA COLLATION ANALYSIS

• Commercial Software Data Logger
• Usability Activity Log v2.3
• Noldus Observer
• OVO Logger 4.1
• Techsmith
• Usability Systems/Alucid

40
6.1 Data Logging
6. DATA COLLATION ANALYSIS

• Advantages
• One package fits all
• Accommodate usability data needs in a single tool
• Remote capabilities
• Automatic collection of quantitative data

• Disadvantages
• Expensive
• Complex to learn
• Require significant hard drive space for
  installation
• Time-consuming to use

41
6.1 Data Logging
6. DATA COLLATION ANALYSIS
Usability Datalogger v3.0 Free and easy to use
solution from userfocus.com

• A collection of worksheets contained within a
  single Microsoft Excel file
• Collect general data as well as qualitative and
  quantitative data
• Up to 20 participants, 20 tasks with 12 pre and
  post test questions
• Unlimited note-taking of observational data
• Built-in timer to collect task completion time
• Can preset effectiveness level
• Pre-defined categories for coding data
• Does not provide video link capability
• Does not generate reports

42
6.1 Data Logging
6. DATA COLLATION ANALYSIS
Usability Datalogger

• Output
• Organised summary of pre and post test questions
• Task completion chart for effectiveness
• Time and task chart for efficiency
• Satisfaction chart

43
6.1 Usability Datalogger
6. DATA COLLATION ANALYSIS
About Title screen, credits and basic
information to tool
44
6.1 Usability Datalogger
6. DATA COLLATION ANALYSIS
Admin Test protocol data (participant details,
task order, etc.)
45
6.1 Usability Datalogger
6. DATA COLLATION ANALYSIS
Admin Effectiveness scoring criteria
46
6.1 Usability Datalogger
6. DATA COLLATION ANALYSIS
Tasks and Questions Tasks, pre-test and
post-test questions
47
6. DATA COLLATION ANALYSIS
6.1 Usability Datalogger
Data collection sheets Pilot 1 2 and
Participant 1 20
48
6. DATA COLLATION ANALYSIS
6.1 Usability Datalogger
Satisfaction Manual entry of satisfaction survey
data
49
6. DATA COLLATION ANALYSIS
6.1 Usability Datalogger
Charts Automatic chart generation of
effectiveness,
50
6. DATA COLLATION ANALYSIS
6.1 Usability Datalogger
Charts Automatic chart generation of efficiency
51
6. DATA COLLATION ANALYSIS
6.1 Usability Datalogger
Charts Automatic chart generation of satisfaction
52
6.1 Usability Datalogger
6. DATA COLLATION ANALYSIS
Automatic data collation tabs
Pre-questions Summary of individual pre-question
responses
Observations Summary of individual observations
Post-questions Summary of individual
post-question responses
Task print out Formatted tasks for a report
table and/or a task booklet for the test
DataSum Data summary sheet used to populate
charts worksheet
Read only worksheets with Auto Fit Rows button
to transfer inputted data
53
Further Reading

• Wilson and Corlett, 1999. Evaluation of Human
  Work a practical ergonomics methodology. Taylor
  and Francis. Chapter 3 and 10
• Jordan, 1998. An Introduction to usability.
  Taylor and Francis. Chapter 5
• J. Rubin, 1994. Handbook of Usability Testing
  How to Plan, Design and Conduct Effective Tests.
  Wiley Technical Communication Library
• Oppenheim, 1966. Questionnaire Design and
  Attitude Measurement. LondonHeinemman.
• Edwards, 1957. Techniques of Attitude Scale
  Construction. New YorkAppleton-Century-Crofts
• Kirakowski, J and Corbett, M, 1988, Measuring
  User Satisfaction, in D M Jones and R Winder
  (Eds.) People and Computers IV. Cambridge
  Cambridge University Press.
• Brookes, 1996. A Quick and Dirty Usability Scale.
  http//www.cs.concordia.ca/soen357_4/SUS-paper.pd
  f
• Reason, 1990. Human Error. Cambridge.

54
Useful Websites

• www.userfocus.co.uk
• www.deyalexander.com.au
• www.upassoc.org
• www.hfes.org
• www.ergonomics.org.uk
• www.ergoweb.com

55
Any Questions?
Thank you
Nantida Wisawayodhin nantida24_at_gmail.com
56
BREAK 15 MINUTESPRACTICAL SESSION 1.5 hour
57
Practical Session

• Familiarise yourself with product and tasks to be
  performed
• Aim How easy is it to perform the task?
• List out relevant factors
• Outline intended data to be collected
• Design questionnaire/interview/checklist

1045 1125
1125 1215

• Conduct a pilot study
• Use Usability Datalogger to collect and analyse
  data

1215 1230

• Presentation of each step
• Discussion
• Wrap up