Journal

1
Journal pager studies

• Experience sampling techniques

2
Agenda

• Questions
• Examples of physical prototypes
• Exam II format
• Overview of sampling techniques
• Pager studies
• Diary studies

3
Overview of sampling methods

• Can be formative or summative
• Depends on types of questions
• Format of study
• Uses of results
• Qualitative or quantitative methods
• Pager / beeper sampling
• Journal / diary studies

4
Goals

• Remove constraints of more structured methods
• Understand experience in natural settings
• Better understanding of experience over time

5
Beeper study basics

• Page participants at random or regular intervals
• Have them answer questions in context within a
  fixed amount of time

6
Journal study basics

• Users self-report on experiences at regular
  intervals, as desired, or when a significant
  event occurs
• Given specific questions to answer or general
  subjects about which to report

7
Types of beeper sampling methods

• Taken from psychology
• Thought sampling
• Experience sampling method (ESM)
• Descriptive experience sampling

8
Thought sampling

• Uses random paging
• Series of questionnaires to determine mental
  content
• 24 4-point scales for features of thought
• (e.g., detail, color, auditoriness, interior
  monologue)

9
Experience sampling method

• More general than thought-sampling
• Does not necessarily measure thought variables
• Focuses on context of experience
• Where are you? With whom? What are you doing?
• Often uses Likert scales or multiple choice to
  assess experience

10
Example

• iESP (Intels ESM on PalmOS PDAs)
• Alert the participants about the questionnaires
• Capture the participants' responses
• Consolvo Walker

11
Descriptive experience sampling

• Qualitative (no Likert scales!)
• Encourages study participants to develop their
  own descriptive language
• Less constraint on user responses

12
Benefits

• Asks users to report what they were doing before
  the page
• Users less likely to modify behavior
• Can collect data at any time no observer needed
• Can help get at tacit knowledge

13
Limitations

• Certain populations unwilling to carry pager
• Potentially inconvenient
• How much time between page and response?
• Random or timed paging may miss important
  experiences
• Potential for post-hoc rationalization

14
Pager study design factors

• How often to page
• How much time between page and response
• Random or regular paging
• Question content and format

15
Diary / journal studies

• Qualitative method
• Participant creates logs of activities
• Generally gives control of data collection to
  participant
• Also good for collecting data over time

16
Diary study design factors

• Structure of questions
• Record all activities or questions about specific
  activities?
• Prescribed time to log, or left up to
  participant?
• Once a day?
• Every time you use a system?
• Every time a significant event occurs?
• Means of recording
• Writing, web entry, audio, photographs?

17
Benefits of diary studies

• Non-intrusive, non-interrupting
• Participants log at their convenience
• Potential for detailed, interpreted answers
• Not random
• Participants will report most significant
  information

18
Limitations

• Relies on participants to remember to log
• Amount of information will vary with
  participants motivation
• Much time may elapse between experience and
  reporting
• Potential for post-hoc rationalization

19
Example

• PAL diary study
• Fits in ones pocket
• Record when PAL is desirable
• Hayes et al. MobileHCI 04

20
Some advice

• Truongs rule on beeper journal studies
• Make as convenient as possible for participants
• If it seems the evaluators job is to sit back
  and relax while the participants do all the work,
  
• Understand if significant events are missed
• Understand what the participants logged and why

21
Upcoming

• Advanced topics
• Designing for the Web (Wednesday)
• Alternate input / output (Friday)
• UIST
• UbiComp
• Exam II