Scribe4Me Evaluating a Mobile Sound Transcription Tool for the Deaf

1
Scribe4MeEvaluating a Mobile Sound Transcription
Tool for the Deaf

• Tara Matthews, Scott Carter
• U.C. Berkeley EECS
• Carol Pai
• eBay User Experience Design
• Janette Fong
• Microsoft Windows Mobile
• Jennifer Mankoff
• Carnegie Mellon HCI Institute

2
Scribe4Me

• A tool created for people who are deaf or
  hard-of-hearing, to improve awareness of
  sound-based information in any location
• User Needs
• knowing what just happened is a concern for the
  deaf outside the home office
• both speech non-speech sounds are important
• Prototype mobile tool provides text
  transcriptions of the last 30 seconds of audio,
  upon user request

3
Contributions

• Demonstrate Scribe4Me is feasible valuable to
  users in field study
• Highlight situations in which Scribe4Me is
• useful (many! conversations, announcements,
  machines with voices,)
• limited (face-to-face conversations, private
  speech, loud environments, multiple
  conversations)
• Provide recommendations on useful info for
  transcriptions
• speech
• descriptions of speaker voices (e.g., gender,
  emotion)
• environmental sounds

4
Outline

• Understanding user needs
• Scribe4Me design implementation
• Field study
• Design implications for future mobile sound tools

5
Understanding User Needs Existing Technology

• Assistive technology for the deaf includes
  support for

6
Understanding User Needs Existing Technology

• Assistive technology for the deaf includes
  support for

7
Understanding User Needs Existing Technology

• Assistive technology for the deaf includes
  support for

8
Understanding User Needs Existing Technology

• Assistive technology for the deaf includes
  support for

9
Understanding User Needs Adoption Retention

• Published rates of assistive tech discontinuance
  8 - 75 Garber Gregorio 90 Gitlin 95
  Phillips Zhao 93 Tewey, Baranicle Perr 94
• Making tools usable is a first step to user
  retention
• Longer studies needed to understand adoption
  retention issues

10
Understanding User Needs Interviews

• Interviewed 8 people who are deaf Matthews et
  al. 06
• goal deeper understanding of peoples info needs
  in all the places they go, especially non-speech
  info
• sound awareness important in all locations not
  adequately supported by existing technology
• need awareness of both speech and non-speech
  sounds

Out AboutIf Im in a group and we go to a
loud place, I will withdraw from the conversation
because I cant really follow the conversation
very easily.
HomeI wish I could hear what my kids are saying
when they play together.
DrivingWhen there is something wrong with the
car it tends to go unnoticed until it is very
expensive to fix.
11
Understanding User Needs Pilot Study

• Goal proof of concept is transcribing recent
  sounds valuable and does it make sense to users?
• Participants 2 deaf, 2 hearing (listening to
  music)
• Activities catching bus at station, shopping at
  mall or grocery store
• WOZ method
• researcher followed user
• user pushed button on mobile device when in need
  of sound awareness
• researcher handed user paper-written
  transcription of recent sounds
• Results
• all users found a valuable use (e.g.,
  announcements, communicating)
• wanted increased detail in transcriptions,
  especially exact dialogue

12
Outline

• Understanding user needs
• Scribe4Me design implementation
• Field study
• Implications for future mobile sound tools

13
Scribe4Me Design
1) A participant needs sound info clicks
what happened? button.
14
Scribe4Me Design
1) A participant needs sound info clicks
what happened? button.
2) The past 30 seconds of audio is sent to
the desktop transcription interface.
A person transcribes the audio and
sends it to the participant.
15
Scribe4Me Design
1) A participant needs sound info clicks
what happened? button.
2) The past 30 seconds of audio is sent to
the desktop transcription interface.
A person transcribes the audio and
sends it to the participant.
3) The participant receives the
transcription as a text message.
16
Scribe4Me Implementation

• Mobile Device
• used Momento Mobile Client Carter Ph.D. Thesis
  06 customized UI configured to capture
  transmit audio
• users asked to take a photo with requests
• sends two 15-second WAV files (8KHz mono, 115 kb
  each)
• 2-4 minutes to transmit via GPRS
• devices I-mate PDA2K, Windows Mobile system

17
Scribe4Me Implementation

• Mobile Device
• used Momento Mobile Client Carter Ph.D. Thesis
  06 customized UI configured to capture
  transmit audio
• users asked to take a photo with requests
• sends two 15-second WAV files (8KHz mono, 115 kb
  each)
• 2-4 minutes to transmit via GPRS
• devices I-mate PDA2K, Windows Mobile system

18
Scribe4Me Implementation

• Mobile Device
• used Momento Mobile Client Carter Ph.D. Thesis
  06 customized UI configured to capture
  transmit audio
• users asked to take a photo with requests
• sends two 15-second WAV files (8KHz mono, 115 kb
  each)
• 2-4 minutes to transmit via GPRS
• devices I-mate PDA2K, Windows Mobile system

19
Scribe4Me Implementation

• Mobile Device
• used Momento Mobile Client Carter Ph.D. Thesis
  06 customized UI configured to capture
  transmit audio
• users asked to take a photo with requests
• sends two 15-second WAV files (8KHz mono, 115 kb
  each)
• 2-4 minutes to transmit via GPRS
• devices I-mate PDA2K, Windows Mobile system

20
Scribe4Me Implementation

• Mobile Device
• used Momento Mobile Client Carter Ph.D. Thesis
  06 customized UI configured to capture
  transmit audio
• users asked to take a photo with requests
• sends two 15-second WAV files (8KHz mono, 115 kb
  each)
• 2-4 minutes to transmit via GPRS
• devices I-mate PDA2K, Windows Mobile system

21
Scribe4Me Implementation

• Momento Desktop Transcription System Carter
  Ph.D. Thesis 06

22
Scribe4Me Implementation

• Momento Desktop Transcription System Carter
  Ph.D. Thesis 06

type transcription here
23
Outline

• Understanding user needs
• Scribe4Me design implementation
• Field study
• Implications for future mobile sound tools

24
Field Study

• Study
• 2-weeks, used working Scribe4Me prototype
• Participants (6)
• hearing 2 profoundly deaf, 1 minimal hearing, 3
  some hearing with aids or implants
• location 2 in Pittsburgh area, 4 in San
  Francisco area
• occupations teacher/student, info services
  worker, clerk, small business owner, marketing
  director, and student
• Data
• 4 interviews per user (start, day 4, day 8, end)
• daily email diary
• requests (audio, photo, transcription)

25
Field Study transcriptions

• Grocery
• ltbeepsgt (digital female voice) Move your
  brussels sprouts to the belt 3 please. Thank you
  for shopping at Giant Eagle. Please remember to
  get your receipt. ltbeepinggt

26
Field Study transcriptions

• Airport
• (female) Good afternoon ladies and gentlemen,
  we do apologize for our late start today thank
  you for your patience. We'd like to begin
  boarding... with us today in the 1st class cabin
  on 1215 service to Chicago. If you are traveling
  1st class you are welcome to board. Each
  passenger may carry on 2 items. One is a smaller
  item such as a briefcase or purse. The other
  would fit under the seat in front of you...

27
Results
Average Requests per Day
Total number of requests 118
28
Results
Average Requests per Day
Total number of requests 118
29
Results
Average Requests per Day
profoundly deaf
min hearing
some hearing
30
Results valuable in multiple situations

• (13) Public speech
• others talking
• (33) Semi-public speech
• class, workplace chatter, church
• (20) Conversations in which the user was
  involved
• (14) Announcements in public places
• airport, deli, train
• (16) Using machines with voices
• TV, radio, grocery, fax
• (15) Environmental sounds
• traffic, home, workplace

31
Results valuable in multiple situations

• (13) Public speech
• others talking
• (33) Semi-public speech
• class, workplace chatter, church
• (20) Conversations in which the user was
  involved
• (14) Announcements in public places
• airport, deli, train
• (16) Using machines with voices
• TV, radio, grocery, fax
• (15) Environmental sounds
• traffic, home, workplace

32
Results valuable in multiple situations

• (33) Semi-public speech
• class, workplace chatter, church
• (20) Conversations in which the user was
  involved
• This is great!!!!  We had a group of friends
  over and were playing cards.  All hearing
  (except me). About 1/2 of them sign. I was
  teasing my GF about still having the Xmas tree up
  almost 3 weeks after Xmas.   The "scroooge" part
  was definitely not translated to my memory during
  the teasing (I could have missed it though). 
  Made me crack up to read your translation.  (I
  am a scrooge!)

33
Results situations

• Public speech (13)
• I was just curious what was being said.  There
  were three women talking with a street vendor.

34
Results situations

• Semi-public speech (33)
• The meeting was over and various people were
  chatting. The transcription was useful after
  the fact, since this type of conversational
  situation is the most challenging for me. I was
  appreciative that the tool was available to
  fill in the large gaping holes of conversation
  in a group I usually miss.

35
Results situations

• Conversations (20)
• The tool is most useful conversation-wise.
  Usually I dont really get all that much of the
  conversation and subsist on a few words here
  and there. There were several times I used the
  tool and it may have taken a couple messages and
  a couple minutes but I was privy to the complete
  sentence, which was very nice.

36
Results situations

• Announcements in public places (14)
• I was in the airport. It is hard to tell
  when something important is being announced or
  something relevant to your flight.

37
Results situations

• Using machines with voices (16)
• I was at the grocery store, in the
  self-check out aisle. I really like using the
  tool because the stupid machine doesnt use
  captions on its screens

38
Results situations

• Environmental sounds (15)
• I was trekking home walking Just trying to
  get a gist of my audible surroundings.

39
Results adoption retention

• All participants wanted to continue using
  Scribe4Me in their daily lives
• 5 of 6 users adopted Scribe4Me during the study
• 1 user only tried the tool twice and then gave up
• we gave no incentive to users for using tool
• Finding a valuable use bolstered usage

40
Results technical issues

• Response delay (3-5 minutes)
• Three to five minutes delay leaves one as an
  observer rather than an active participant.
• causes slow existing GPRS networks,
  transcription time
• solutions faster 4G networks, open WiFi networks
• Incomplete / inaccurate transcriptions
• causes imperfect audio quality, dropped packets,
  lack of context
• solutions high-quality microphones, more
  reliable networks

41
Results privacy

• Ianchello et al. 06 people prefer a priori
  informed consent of voice recording replay to
  others (legally required in 12 states)
• Privacy concern (46)
• public speech
• semi-public speech
• No / limited privacy concern (65)
• conversations in which the user was involved
• announcements in public places
• using machines with voices
• environmental sounds

42
Outline

• Understanding user needs
• Scribe4Me design implementation
• Field study
• Design implications for future mobile sound tools

43
Design Implications mobile sound transcribing

• Social limitations
• face-to-face conversation delay social
  expectations (eye contact attention)
• informed consent
• Transcription limitations
• loud environments (e.g., restaurants)
• multiple conversations (e.g., in a classroom)
• Valuable info for transcriptions
• speech
• descriptions of speaker voices (e.g., gender,
  emotion)
• environmental sounds
• Lower cost human-in-the-loop systems

44
Conclusions

• Created Scribe4Me, a tool for people who are deaf
  or hard-of-hearing, to improve awareness of any
  sound-based information in any location
• Demonstrated that Scribe4Me is feasible
  valuable to users
• Highlighted design implications based on field
  study use
• Longer study needed to understand adoption
  retention issues

45
Questions?

• For more information on Scribe4Me, sound
  transcription tool for the deaf
• tmatthew_at_cs.berkeley.edu
• For more information on Momento, prototyping
  tool
• m0ment0.com or sacarter_at_cs.berkeley.edu
• This project was funded by the NSF IIS-0205644,
  IIS-0501895