Journal and Pager Studies plus Empirical Evaluation cont'

1
Journal and Pager Studies(plus Empirical
Evaluation cont.)

• Experience sampling techniques

2
Agenda

• Questions
• Reminders
• Video scenario sample
• Empirical evaluation (cont.)
• Overview of sampling techniques
• Pager studies
• Diary studies

3
Tentative Part 3 schedule

• 7/7 Show me your evaluation plans
• 7/9 Submit IRB protocols
• 7/14 in-class evaluations with HCI experts
• 7/19 Project Part 3 writeups due
• 7/19, 7/21, 7/23? Project presentations
• Im out of town from 7/12-7/18. See Ed for
  assistance and feedback!

4
Video scenario sample

• Spring 2003 - Teachers keeping track of kids at
  the zoo
• This video shows lots of important things like
  what?

5
Empirical Methods

• Controlled laboratory experiment to support a
  claim or hypothesis

6
The Case for Experimentation Benefits
Limitations

• Consider an example
• Designing a website for older adults

7
Benefits

• High degree of control
• Precise measurement (related to control)
• Allows for causal judgments

High Perceptibility (WhizBang.com)
Low perceptibility (BoringStuff.com)
8
Limitations

• Artificiality
• Environment
• Methods
• Ethics

Just be comfortable and respond normally!
9
Designing an empirical study

• Question
• Experimental factors

10
Dont Forget Your Question

• What is your question?
• Review the literature
• Has you question already been answered?
• Is there a debate about your question?
• How have previous studies been designed to answer
  this question?

11
Experimental Factors

• Define the variables
• Hypotheses
• Method
• Participants

12
Define the Variables

• Independent variables Manipulated, Cause
• Dependent variables Measured, Effect
• The dependent variable(s) needs to be affected by
  the independent variable(s).
• Control group

13
Hypotheses

• Hypothesis A variation in the independent
  variable will cause a difference in the dependent
  variable.
• Null hypothesis There is no difference in the
  dependent variable by the independent variable.

14
Example
2 (age) x 4 (site design)
DV Time to complete the task
15
Experimental Method

• Between-groups
• Within-groups
• Mixed

16
Between-groups

• Each participant assigned to a different
  condition
• No learning effects
• -- Disadv. Need more participants
• -- Disadv Individual differences could bias the
  results

17
Within-groups

• Participant tested on all conditions
• Adv. Fewer participants
• -- Disadv. Potential learning effects

18
Mixed

• One variable between-groups and one variable
  within-groups

19
Participants

• Characteristics
• Number
• Consent
• Anonymity
• Risks
• Compensation
• Free will

20
Independent vs. Confound Variables

• Independent variables
• Confound variables
• Participant variables
• Procedural variables

21
Tips

• Experimental control, not as easy as it may seem
• Plan ahead
• Create a detailed protocol be organized
• Pilot studies

22
Evaluation through monitoring physiological
responses

• Eye tracking
• Heart activity blood pressure, volume, pulse
• Galvanic skin response
• Electrical activity in muscles
• Electrical activity in brain

23
Eye tracking
24
Physiological responses

• Can get at tacit responses
• Can somewhat measure interest, involvement,
  emotional response, stress

25
Overview of sampling methods

• Can be formative or summative
• Depends on types of questions
• Format of study
• Uses of results
• Qualitative or quantitative methods
• Pager/beeper sampling
• Journal/diary studies

26
Goals

• Understand experience in natural settings
• Better understanding of experience over time
• Remove constraints of more structured methods

27
Beeper study basics

• Page participants at random or regular intervals
• Have them answer questions in context within a
  fixed amount of time

28
Journal study basics

• Users self-report on experiences at regular
  intervals or as desired
• Given specific questions to answer or general
  subjects about which to report

29
Types of beeper sampling methods

• Taken from psychology
• Thought sampling
• Experience sampling method (ESM)
• Descriptive experience sampling

30
Benefits of beeper sampling
31
Thought Sampling

• Uses random paging
• Series of questionnaires to determine mental
  content
• 24 4-point scales for features of thought
• (eg. Detail, color, auditoriness, interior
  monologue)

32
Experience Sampling Method

• More general than thought-sampling
• Does not necessarily measure thought variables
• Focuses on context of experience
• Where are you? With whom? What are you doing?
• Often uses Likert scales or multiple choice to
  assess experience

33
Descriptive Experience Sampling

• Qualitative (no Likert scales!)
• Encourages study participants to develop their
  own descriptive language
• Less constraint on user responses

34
Limitations

• Certain populations unwilling to carry pager
• How much time between page and response?
• Potential for post-hoc rationalization
• Potentially inconvenient
• Random or timed paging may miss important
  experiences

35
Pager study design factors

• How often to page
• How much time between page and response
• Random or regular paging
• Question content and format

36
Benefits

• Asks users to report what they were doing before
  the page
• Users less likely to modify behavior
• Can collect data at any time no observer needed
• Can help get at tacit knowledge

37
Diary/Journal studies

• Qualitative method
• Participant creates logs of activities
• Generally gives control of data collection to
  participant
• Also good for collecting data over time

38
Diary study design factors

• Structure of questions
• Record all activities or questions about specific
  activities?
• Prescribed time to log, or left up to
  participant?
• Once a day? Every time you use a system?
• Means of recording
• Writing, web entry, audio, photographs?

39
Benefits of diary studies

• Non-intrusive, non-interrupting
• Participants log at their convenience
• Potential for detailed, interpreted answers
• Not random
• participants will report most significant
  information

40
Limitations

• Relies on participants to remember to log
• Amount of information will vary with
  participants motivation
• Much time may elapse between experience and
  reporting
• Potential for post-hoc rationalization

41
Upcoming

• Monday Ubicomp
• Weiser article on web be prepared to discuss