experience sampling method: preliminary findings

1
experience sampling method preliminary findings

• sunny consolvo miriam walker
• intel research seattle
• sunny_at_intel-research.net
• user-centered evaluations of ubicomp workshop
• 4th intl conference on ubiquitous computing
• september 29, 2002

2
before we get started

• this work is preliminary
• wrapped up data collection this month
• expect future publication

3
outline of talk

• intro to experience sampling method (esm)
• case study the personal server
• issues
• wrap-up

4
esm a brief introduction

• also called
• beeper studies
• time sampling
• field study technique from psychology
• used to understand areas such as
• mood
• time use
• social interactions
• were exploring its uses for ubicomp

5
esm a brief introduction

• fill out questionnaire when alerted
• researcher is not present
• alerted several times per day
• business as usual until alerted
• duration is often 1-3 weeks

6
running esm tools

• need to choose tools for
• alerting participant
• random alerts
• timed alerts
• event-based/user-driven
• delivering questionnaire
• capturing participant responses

7
choosing tools alerting

• tools for alerting the participant include
• cell phones
• pagers
• watches
• PDAs
• custom devices
• some desirable abilities
• specify time period during which alerts may go
  off
• specify of alerts per time period overall

8
choosing tools delivering

• tools for delivering questionnaires include
• paper booklets
• PDAs
• phones (traditional or mobile)
• live researcher
• recorded message
• audio recordings (e.g., cassette player)

9
choosing tools capturing

• tools for capturing responses include
• paper booklets
• PDAs
• phones (traditional or mobile)
• live researcher
• recording service (e.g., Tellme)
• audio recordings (e.g., cassette player)
• cameras

10
esm some advantages

• reduces effects of intrusive observers
• larger sample size time interval
• reduces memory biases by asking about current
  activity, feelings, etc.
• its all about now

11
esm more advantages

• less time consuming than observations, yet
• study activities as they occur in the field
• study activities and emotions often not
  accessible to observers
• not as disruptive to others

12
esm some disadvantages

• questionnaires must be brief
• participant must carry tools at all times
• power consumption and/or service availability
• certain events may only show up in very large
  samples sizes/durations
• more intrusive than interviews surveys

13
outline of talk

• intro to experience sampling method (esm)
• case study personal server
• issues
• wrap-up

14
case study personal server

• personal server team was interested in
• how people use information
• what they want
• when they want it
• where they want it
• used esm to help collect this data

15
pilot study 1

• 4 participants
• lab researchers administrative staff
• palm m100s for
• alerting
• delivering
• capturing
• original boston college ESP software
• for running ESM on palms
• duration 1½ - 4 days

16
pilot study 1 problems

• palm dropped ? batteries fell out ? data lost
• only method of alert was beep (m100)
• participants wouldnt carry it in several
  situations
• once volume of beep was set, it couldnt be
  changed
• only response option multi-choice, single
  answer
• difficult for participants to answer some
  questions
• e.g., where are you?
• some participants wanted a bag to carry the palm
• participants wanted to know about missed
  questionnaires

17
pilot study 2

• 4 participants
• lab researchers, administrative staff, family
• modified iESP software
• palm m500s for alerting and delivering
• for capturing
• 2 participants carried fold-out keyboards (for
  palm)
• 2 participants carried paper booklets
• all 4 carried digital cameras
• for describing where what they were doing
• duration 1½ - 4 days

18
pilot study 2 justifications

• upgrading to m500s
• ability to alert via beep and/or vibrate
• almost impossible for batteries to fall out
• iESP software added functionality, including
• freeform text entry
• multi-choice, multi-answer
• new formats for multi-choice, single answer
• slider (e.g., on a scale of 1 to 5)
• ability to change volume, vibrate, etc. during
  study
• ability to check how many questionnaires have
  been completed vs. asked during study

19
our implementation the study

• 31 participants
• recruited from local community
• pre/post-study interviews 1 week with equipment
• 10 questionnaires/day
• 70 questionnaires total
• incentive (in USD)
• 50 for participating and returning equipment
• 1 per completed questionnaire
• 120 total possible

20
the study (continued)

• palm m500 to alert, deliver, capture
• modified iESP software
• more about capturing
• also used camera
• free-form data entry
• paper booklet for all 31 participants
• graffiti / soft keyboard for all 31 participants
• full-sized fold-out keyboard for 9 of 31
  participants

21
what the participants carried
22
type of data collected

• responses to following types of questions
• yes / no
• multiple choice, single answer
• multiple choice, multiple answers
• freeform text entry
• looks a lot like survey data except
• multiple responses from same participant in
  variety of circumstances over time

23
other data collected

• qualitative data from interviews
• participants photos
• some photos were of interest to the participant
• other photos documented where s/he was or what
  s/he was doing

24
questions on the palm
25
more questions
26
participant photos in transit
27
participant photos at home
28
participant photos at work
29
participant photos retail
30
participant photos pets
31
outline of talk

• intro to experience sampling method (esm)
• case study personal server
• issues
• wrap-up

32
our tool decisions

• chose palm because
• no service problemscost or coverage
• power consumption not an issue for duration
• know when participant fills out questionnaire
• probabilities and contingencies can be used
• if a, then b
• if b, then not c
• d 50 of time
• dont have to transcribe data

33
tool decisions

• used camera to see if it would help participants
  describe
• where they were
• what they were doing
• photos used to jog memories during post-study
  interviews

34
according to the participants

• 1/31 would not participate again
• 2/31 would only do it for more money
• 28/31 would participate again
• it was cool
• I felt like a spy
• friends were jealous of the fold-out keyboard
• a lot to carry
• mixed feelings about carry bag
• several claimed they would be purchasing palms
• only one participant wanted voice recording
  option
• when asked, most did not respond favorably

35
from our perspective

• woo-hoo! we got all of the equipment back
• only 1/31 palms was broken
• of the 31 participants and 70 questionnaires
• 28 participants completed 50 questionnaires
• 13 of whom completed 60 questionnaires
• 2 participants completed 25- questionnaires
• 1 participant performed a hard reset and returned
  the palm with no data or software

36
outline of talk

• intro to experience sampling method (esm)
• case study personal server
• issues
• wrap-up

37
wrap up well use it again

• pleased with results so far
• intend to use esm again

38
wrap-up pda software

• original ESP software available at
• http//www2.bc.edu/barretli/esp/
• iESP software (includes upgrades)
• launching on intel research seattle web site in
  Oct or Nov 2002
• if you want to be notified
• business card with iESP notification written on
  it
• legibly fill out notification sign-up sheet

39
acknowledgements

• drs. lisa daniel barrett
• justin silves
• gaetano borriello
• ken anderson
• rahul shah
• trevor paring
• roy want
• john light
• our pilot testers
• and our colleagues at intel research seattle

40
questions?
41
examples of study questions

• where is your cell phone? in my
• hand
• pocket/belt clip
• bag
• car
• home
• office
• somewhere else and with me
• somewhere else and not with me

42
examples of study questions

• in the last 30 minutes, which of the following
  would you have liked to have access to but didnt
  have available?
• web pages
• movie times
• restaurant reviews
• personal calendar
• telephone number(s)
• computer files
• music files
• shopping list
• to do list