A Mixed Method for Common Input Devices Evaluation

1
A Mixed Method for Common Input Devices
Evaluation

• Murni Mahmud

2
Outline

• Research Overview
• Research in input devices
• Understand users and their abilities
• Research questions and objectives
• Experimental Design
• Design and procedures
• Results
• The users
• Relationship between ability and performance
• Perception and attitude
• User behaviour model and microstrategies
• Summary
• Conclusions
• Contributions
• Limitations and future work

3
Research Background

• Input Devices Evaluation
• Most studies use the Fitts' Law task as
  recommended by ISO-9241.
• Fitts' Law tasks are experimental tasks, which
  sometimes may not represent realistic settings.
• Many research in input device evaluation focus on
  specific group of users children, adult or older
  adults users.
• Knowing the user
• Analyse user characteristic general abilities,
  physical abilities and skills (Sutcliffe, A.
  1995).
• Model Human Processor (MHP) consider 3 abilities
  for information processing cognitive, motor and
  perceptual in MHP (Card, Moran Newell, 1983).
• User sensitive inclusive design (Newell, A. F.
  Gregor, P., 2000).
• Psychometric Tests
• Assessment tool to measure sensitivity, memory,
  intelligence, aptitude or personality, for
  example the Kit of Factor-Referenced Cognitive
  Tests, the Bender Visual Motor Gestalt test and
  Differential Ability Scores.
• Psychometric Test in HCI influence of perceptual
  speed in information retrieval (Allen, 1994) and
  influence of ability in data entry task (Czaja et
  al., 1998).

4
Questions and Objectives

• Research Questions
• What are the relationships between user ability
  and task performance for selected input devices
  in different contexts of use?
• What are user perceptions of usability of the
  selected input devices in different contexts of
  use?
• What is the user behaviour when using input
  devices and does it influence task performance?
• Research Objectives
• To investigate if there is any correlation
  between user ability, user perception and user
  task performance when using input devices in
  different contexts of use.
• To study if a particular input device is superior
  in terms of task performance and user perception
  in a particular context of use. In specific, what
  are the qualities that make one input device
  superior to another.
• To investigate user behaviour when using input
  devices for different tasks.

5
Experimental Design

• Demographics questionnaire
• Elicit subject background, experience using the
  internet, computers and input devices.
• Psychometric Tests
• Mini Mental State Exam (MMSE) for cognitive
  ability.
• Identical Pictures (IP) for perceptual speed.
• Simple Reaction Time (SRT) for motor ability.
• Tasks and Input Devices
• Browsing a website (browsing point and select).
• Playing a solitaire card game (dragging point,
  drag and drop).
• Input Devices Mouse, Tablet-with-Stylus, Touch
  Screen.
• Pre-task and Post-task Rating
• Ease of use and difficulty ratings (5 liked
  scale).
• Interview
• Elicit user justification for ratings and
  experience using input devices.

6
Tasks and Input Devices

• Browsing Dragging

Mouse Tablet-with-Stylus Touch Screen
7
Psychometric Tests

• Cognitive MMSE

• Naming
• Repetition
• Comprehension
• Reading
• Writing
• Drawing

• Time Orientation
• Place Orientation
• Registration of Three Words
• Serial 7s As a Test of Attention and Calculation
  
• Recall of three words

• Motor SRT

• Perceptual Identical Picture

8
Results User Ability and Performance

• Demographics
• 60 participants, age from 12 to 75 years M30.6,
  SD18.19.
• Computer/Internet Experience gt 2years (6 months
  30 years)
• Device Experience Mouse (M8.1,SD5.75)
  Tablet-with-stylus (M0.4, SD0.97) Touch
  Screen (M2.0, SD2.26)
• User Ability
• MMSE a stable cognitive ability (Med28.23).
• IP a reasonably high perceptual speed score
  (80.66).
• SRT better completion time when using a Mouse
  (281ms), followed by a touch screen (362ms) and a
  Tablet (378ms).
• Significant correlation perceptual speed and SRT
• User Performance Comparison
• Browsing NS - (1) Touch Screen (2) Mouse (3)
  Tablet
• Dragging SIG (1) Mouse (2) Touch Screen (3)
  Tablet
• Relationship between User ability and Performance
• Significant correlation perceptual speed, motor
  and user performance in dragging task.

9
Results User Perception

• Ratings
• Pre-task (1) Mouse (2) Touch screen (3) Tablet
• Post-task Improved (1) Mouse (2) Touch screen
  (3) Tablet
• Browsing (ease of use) (1) Mouse (2) Touch
  screen (3) Tablet
• Dragging (ease of use) (1) Mouse (2) Tablet (3)
  Touch screen
• General task (ease of use) (1) Mouse (2) Touch
  screen (3) Tablet

• Qualities of Input Devices

• Attitude
• ve Mouse Effectiveness, Ease of use,
  Familiarity
• -ve Tablet Effectiveness, Physical Environment
• ve Touch Screen Accessibility and Familiarity

10
Results User BehaviourSimplistic Models
Dragging
Browsing
11
Partial Models
Browsing
Dragging
12
Structured Models
Browsing
Dragging
13
Results Microstrategies
14
Results Microstrategies
15
Results Microstrategies
16
Summary Measures forInput Devices Evaluation
Objective Measurements

• Quality and Attitude (content analysis)
• Issues in evaluation (content analysis)

• Ability (Psychometrics Tests)
• Performance (time, steps, errors)
• Behaviour (Microstrategies)

Quantitative Approach
Qualitative Approach
Subjective Measurements

• Perception (pre-task and post-task ratings and
  preferences)
• Health (self-reported)

• Self-reported Behaviour Model (State-diagram)

Contexts Input Devices and Tasks
17
Conclusions

• Q1 To identify the relationship between user
  ability and task performance.
• User motor and perceptual ability are correlated
  with user performance in the dragging task.
• Q2 To study the user perception of usability of
  input devices.
• Mouse was perceived as the easiest device for
  browsing, dragging and general tasks.
• Tablet-with-stylus was considered as the most
  difficult for browsing and general task.
• Touch screen was found as the most difficult for
  dragging.
• Qualities Effectiveness, Ease of Use,
  Familiarity, Satisfaction, Physical Environment,
  Efficiency, Accessibility
• Q3 To investigate the users behaviour.
• Self reported behaviour model in state diagram
  Simplistic, Partial and Structured.
• Low-level behaviour microstrategies for SRT,
  First click and first drag for all devices.

18
Contributions

• Augment input device evaluation method
• Incorporates psychometric tests to measure user
  ability where a specific certification is not
  required.
• Introduces real tasks in the natural working
  environment for evaluating input devices to
  obtain objective performance measure.
• Elicits pre-task and post-task subjective rating
  to obtain subjective performance measure, elicits
  user justification
• Conducts an evaluation of input devices with wide
  range of participant age.
• Produce empirical evidences
• Successful use of common psychometric tests to
  measure ability robust SRT test
• Significant correlation between user ability and
  user performance in using input devices
  perceptual and motor for dragging task.
• Consistency between objective measures and
  subjective measures where subjective measures
  explain and support the objective measures.

19
Contributions (cont.)

• Qualities of input devices
• show that effectiveness, ease of use and
  familiarity are the top three qualities of input
  devices evaluation in a wider user population.
• indicate that attitude toward the quality
  reflects the user perception of the input
  devices.
• User behaviour models and microstrategies
• enhance a descriptive model of user behaviour of
  using input devices which were derived from
  self-reported procedure simplistic, partial and
  structured.
• extend slow and fast microstrategies for a
  tablet-with-stylus and a touch screen for a
  click, the first point-and-select of browsing
  task and the first point-select-drag-drop of
  dragging task.

20
Limitations and Future Work

• Limitations
• Deliberate selection of input devices only a
  mouse, a tablet-with-stylus and a touch screen.
• Selection of psychometric tests MMSE is good for
  screening but less effective for measuring user
  cognitive ability.
• Future Work
• Applying the mixed method for evaluating more
  input devices when the influence of ability is
  important devices for people with impairment,
  advance input devices such as body tracking
  device in virtual reality.
• Using the mixed method as a general framework for
  evaluation which could be applied in any
  interface or other devices evaluation.