Usability Study Design

1
Usability Study Design

• John C. Tang
• November 8, 2007

2
New assignment!

• Travel to an unfamiliar country
• Alone
• Especially if youre unfamiliar with the language
• Record your first impressions
• (only kidding about being an assignment!)

3
Conceptual Model of a System

• Design Model
• The model the designer has of how the system
  works
• System Image
• How the system actually works
• The structure and behavior of the system
• Users Model
• How the user understands how the system works

4
Mismatched conceptual models
?
?
?
5
Flying from Shanghai to Jinan
6
(No Transcript)
7
(No Transcript)
8
Hongqiao vs. Pudong airports
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Current events

• OpenSocial APIs
• Profile Information (user data)
• Friends Information (social graph)
• Activities (things that happen, news feed
  type stuff)
• Focus on concepts, not implementation

17
Current events (2)

• Facebook ads
• Business facebook pages
• Spreading marketing messages through social
  connections
• Allowing businesses to gather facebook activity
  data
• Further diluting the meaning of friend?
• Beware social phishing

18
Today

• Review usability study methods
• Apply to facebook applications
• Review actual usability study report

19
Methods for evaluating system

• Qualitative
• Rich, subjective
• Exploring concepts
• Quantitative
• Precise, objective, repeatable
• Demonstrating claims

20
Critical incident technique

• Noting significant events that affect, either
  positively or negatively, the users ability to
  accomplish a task
• Gather incidents over participants, look for
  trends, patterns, correlations

21
Questionnaires surveys

• Design questions with analysis in mind
• Closed format more precise, easier to analyze
• Convert qualitative?quantitative measures
• You give categories to users
• Open-ended questions provide richer feedback,
  longer to analyze
• Users give you categories

22
Designing survey questions

• Multiple choice
• Collecting information
• Ordinal ranking
• Expressing relative preferences
• Likert scales
• Expressing personal reactions

23
Likert scales

• Ask users to rate on a numeric scale
• Odd number scale allows a neutral midpoint (5- or
  7-point scale)
• Even number scale forces taking a position (4- or
  6-point scale)
• Anchors give examples of points along the scale

24
Semi-structured interviews

• Interactively asking questions (face-to-face,
  telephone)
• Give users chance to explain why to complement
  what they did, subjective users viewpoint
• Can help with design questions
• What improvements would you suggest?
• Can be done individually or in groups

25
Quantitative measures (comparison)

• Independent variables
• Attributes we manipulate / vary in condition
• Levels, value of attribute
• Dependent variables
• Outcome of experiment, measures to evaluate
• Usually measure user performance
• Time to completion
• Errors
• Amount of production
• Measures of satisfaction
• Flow path tracking

26
Experiment design (2)

• Control variables
• Attributes that remain the same across conditions
• Random variables
• Attributes that are randomly sampled
• Can be used to increase generalizability
• Avoiding confounds
• Confounds are attributes that changed but were
  not accounted for
• Confounds prevent drawing conclusions on
  independent variables

27
Flow tracking

• Track sequence of pages / steps visited to detect
  typical flow through application
• Look for opportunities to optimize, simplify flow

28
Usage logging

• Embed logging mechanisms into code
• Study usage in actual deployment
• facebook usage metrics
• Anonymous data from unknown users
• Could embed survey mechanism in application

29
Combining methods

• Usage logging for breadth
• Interviews for depth
• Rating perceptions on scales for measure
• Interview for understanding

30
Cardinal rule

• Pilot test your study!
• Feels like youre throwing away data, but youre
  saving time by ensuring proper data collection
• Example I recently lost data because usage
  logging mechanism not sufficiently tested

Go Stanford!
31
Recording the screen

• Recording data for usability study
• Recording demo videos
• Camtasia by TechSmith (for Windows)
• http//techsmith.com/camtasia.asp

32
Camtasia

• Enables recording video of computer screen plus
  audio
• Software install on target computer
• Configure capture frame rate to ease loading on
  the computer
• 30-day free trial

33
Camtasia demo
34
Evaluating facebook applications?

• Measures of perception may be most meaningful
• Hard to find meaningful performance metrics
• Asking you to specify metrics you would monitor
  over long term deployment
• Popularity ? Usability

35
facebook demos

• New York Times quiz
• Scrabulous

36
For your project

• Require aspects of both qualitative and
  quantitative methods
• Qualitative
• How users react to application, perceptions
• Quantitative
• How users perform on application, ratings
• What would you improve on next iteration?
• Perhaps users perceptions of performance more
  important than actual values
• User study of at least number of members in group
• Volunteer for other groups, but no mutual testing
• Examples from Fall 2006 http//vis.berkeley.edu/c
  ourses/cs160-fa06/wiki/index.php/Pilot_Usability_S
  tudy

37
Consent form

• If doing recording of data, have users sign a
  consent form
• Ask yourself, If my data were being recorded,
  what would I want to know?

38
Contact info for questions
How data will be used
What activity observed
What data collected
How to Delete data
Who can access data
Review before show publicly
39
Project status

• Good work on checkpoint!
• Our feedback to you
• Feel free to seek out more input
• Work through platform stability issues
• Use rhombus or pentagon
• facebook pushes are Tuesday evening

40
Planning for the demo!

• Nov. 13 is demo of application
• Intended to be implementation complete
• Rest of time is for usability study and final
  presentation
• Tune demo to show best of application
• Avoid unseemly parts of application
• Create story to show off best parts
• Coding after demo should be limited

41
Presenting your work demo

• Tuesday, 1030-1200noon
• Wozniak Lounge (open starting at 1000)
• Larger community invited (1100)
• Dress business casual
• Refreshments!
• Demo
• Poster

42
Demo

• Prepare, rehearse a demo script
• Introduce need, motivation
• Walk them down a garden path
• Use this to prioritize development work
• Contingency plans
• Screenshots (on poster, stored on machine)
• Video demo (use Camtasia)

43
Poster

• Template sent via email
• Need
• Screenshot(s)
• Indicator of design process
• Minimize text, Maximize interest
• Submit by Sunday, Nov. 11 by midnight to
  cs160_at_imail.eecs.berkeley.edu

44
HelloWorld Application
John Doe Jane Smith Mary Chung Lee Lin
The Hottest Place to Greet the World!
Goal
What need our project fulfills ie. Whats the
problem I solve
Look heres a picture
Callouts to show features
What we did
What does the app do!
Callouts to show features
Ooh Details. Reasons why this is cool
Key design decisions? Why does it do what it does?
Additional Functionality See, click here to do
this!
Iterations of our Design This is how we built it!
In Summary
Our app encourages social interaction!
45
Final Projects CS 160 Fall07
User Interface Design, Prototyping, and Evaluation
Applications
Social Networking
This terms group project was to design and
prototype a facebook application. Students
identified user needs and created designs that
leveraged social interactions. They implemented
on the facebook platform. User study evaluation
will be included in Final Presentation Dec. 4 6
AutoClub
f
BestEats
f
f
Birthday Card
Blurbs
f
Comic Avatar
f
DrawIT
f
f
Headhunters
Pyramid Com-munications Hub
f
Size Me Up
f
Tag Your Friends
f
f
Travelogue
Berkeley IBM the teaching team are volunteers
from IBM Almaden Research Center and Berkeley TAs
John C. Tang, Christine Robson, David Sun, Bryan
Tsao
http//inst.eecs.berkeley.edu/cs160/fa07/
46
Grading criteria

• Need (Is need properly explained? Is this
  useful?)
• Flow through application steps (Are steps
  through the activity clear? Is ordering logical?)
• Usability (Ease of use, Visibility of
  affordances)
• Appearance (Attractive? Good use of color? Fit
  with facebook?)
• Implementation (Did everything work ok?)
• Presentation (Did demo clearly explain
  application? Did poster clearly explain project?)

47
Individual Assessments

• Team dynamics are part of the course
• For each group member (including self)
• Rating from 1 (weak contribution) to 5 (strong
  contribution)
• Paragraph description per person
• Email to cs160_at_imail.eecs.berkeley.edu

48
Usability study report