Current Evaluation Techniques for Mobile Devices

1
Current Evaluation Techniques for Mobile Devices

• Aaron Cooper
• Nov 29, 2004

2
Outline

• Definitions
• Current Techniques and Trends
• 2 Examples
• Future Work

3
Definitions

• Technique
• systematic procedure by which a complex or
  scientific task is accomplished (Dictionary.com)
  
• Evaluation technique
• Procedure involved in assessing an idea
  (product, interface, interaction method, etc.)
• Two categories
• User tasks
• Data collection

4
Techniques currently available

• Desktop Simulations
• Mobile design tested on a PC
• Traditional-only
• Lab test, user is seated
• Heuristic evaluation, observation, think aloud,
  etc.
• Mobile Scenarios
• Motion
• User is physically moving
• Distraction
• Immersed in a primary task while interacting with
  device
• Use traditional data collection methods
• video, questionnaires, interviews
• Must strive for more mobile scenarios

5
Why do we need Mobile Scenarios?

• More context more identified issues
• Effects of cognitive workload (Lai et. al, 2001)
• Environmental Noise (Newcomb et. al, 2003)
• Gesture Suitability (Pirhonen et. al, 2002)
• Added realism
• Walking vs sitting at desk
• Design Appropriateness
• ...so that designers can be sure the devices and
  interfaces they create are usable - Brewster
  (Overcoming), 2002

6
Trends

• Majority use traditional-only techniques
• Field studies are difficult
• Loss of control, unknown variables (Kjeldskov et.
  al, 2004)
• Hard to obtain high-quality video data (Pirhonen
  et. al, 2002)
• No video task monitoring (Newcomb et. al, 2003)
• Lab experiments are the norm
• High-quality data variable control (Kjeldskov
  et. al, 2004)
• Difficulty with realism (Kjeldskov et. al, 2004)
• Very few conducted evaluations specifically
  targeted for a mobile system

7
Mobile Scenario Examples

• Kjeldskov Stage, 2004
• Text messaging
• 2 experiments (motion distraction)
• Compared lab tests against field study
• Brewster (Overcoming), 2002
• Sonically-enhanced PDA buttons
• Pilot study 2 experiments
• Compared lab tests against field study

8
Kjeldskov Stage, 2004 1st Experiment - Motion

• PDA text messaging
• Lab tests
• 5 configurations based on motion
• Field study
• Walking down street
• Data Collection
• Video
• TLX
• Think aloud

9
Kjeldskov Stage, 2004 1st Experiment - Motion
10
Kjeldskov Stage, 2004 2nd Experiment -
Distraction

• Phone text messaging
• Lab test
• Video game with dancing mat
• Video recording
• Field study
• Walking down street
• Audio and written notes

11
Kjeldskov Stage, 2004 Findings

• Experiment 1 (motion)
• Sitting technique good for cosmetic issues
• Layout Navigation suited for motion tests
• Performance similar among techniques
• Track techniques require improvements
• Video for field study was difficult
• Experiment 2 (distraction)
• Video game more attention required
• Audio notes not as useful as video for field
  study

12
Brewster (Overcoming), 2002Pilot Study

• Preliminary Pilot Study
• Lab test, sitting at desk
• Entering of 5-Digit Codes
• Large buttons (16 x 16) vs small buttons (4x4)
• Three different sounds (none, basic, enhanced)
• Results
• Enhanced sounds better or same as basic
• Small buttons were too small
• Pen obstruction and no highlighting
• Increased to 8x8 for experiments

13
Brewster (Overcoming), 2002Prototypes
Small (4x4)
Regular (16x16)
Medium (8x8)
14
Brewster (Overcoming), 2002Experiments

• Experiment 1
• Basic sounds removed
• Lab test (same as pilot study)
• Experiment 2
• Same as above, except performed outside
• Walk from end to end in a quiet path
• Data Collection
• Same as pilot study
• TLX
• Results
• Outside Performance decrease and workload
  increase
• Performance similar, but medium buttons require
  mental effort
• More realistic situation revealed more issues
• Still artificial even more realism required

15
Future Work Challenges

• Easing the conduction of field studies
• Improve data collection
• Incorporate more context
• Seamless integration (Sharples et. al, 2002)
• Social and physical factors (Kjeldskov et. al,
  2004)
• Increase the realism

16
References

• Brewster, S. 2002. Overcoming the lack of screen
  space on mobile computers.
• Personal and Ubiquitous Computing. 6 188205.
• Brewster, S., R. Murray. 2000. Presenting dynamic
  information on mobile computers.
• Personal Technologies. 4 209212.
• Kjeldskov, J., J. Stage. 2004. New techniques for
  usability evaluation of mobile systems.
• International Journal of Human-Computer Studies.
  60 599620.
• Lai, J., K. Cheng, P. Green, O. Tsimhoni. 2001.
  On the road and on the web? Comprehension of
  synthetic and human speech while driving. In
  Proceedings of the SIGCHI conference on Human
  factors in computing systems (March 31 April
  4, Seattle), p. 206212.
• Newcomb, E., T. Pashley, J. Stasko. 2003. Mobile
  Computing in the Retail Arena.
• In Proceedings of CHI 2003 (April 5-10, Ft.
  Lauderdale), p. 337-344.
• Pirhonen, A., S. Brewster, C. Holguin. 2002.
  Gestural and audio metaphors as a means of
  control for mobile devices. In Proceedings of
  the SIGCHI conference on Human factors in
  computing systems (April 20-25, Minnesota), p.
  291-298.
• Sharples, M., D. Corlett, O. Westmancott. 2002.
  The design and implementation of a mobile
  learning resource. Personal and Ubiquitous
  Computing. 6 220234.