The Sound of One Hand: A Wristmounted Bioacoustic Fingertip Gesture Interface

1
The Sound of One HandA Wrist-mounted
Bio-acoustic Fingertip Gesture Interface
Brian Amento, Will Hill ATT Labs
Research Loren Terveen University of Minnesota
2
Outline

• Motivation
• Gesture Interfaces
• Signal Classifiers
• Prototype Applications
• Future Work

3
Motivation

• Small wearable digital devices increasingly
  popular (Cellphones, PDAs, Rios, etc..)
• Nonlinear access to linear media will increase
• Voicemail, Music, Video, Radio, Text
• Controls Device Select, Play, Stop, Scan
  forward, Scan backward, Faster, Slower, Item
  Select, Exit

4
Current Interfaces to Mobile Devices

• Two-handed control mechanisms
• Pressing device buttons
• Writing/selecting with stylus
• Un-holstering a wearable is a pain (i.e.,
  wristwatches beat pocket watches)
• Speech recognition
• Noise or social setting may rule out voice
  control
• Our Goal Invisible, weightless, un-tethered and
  cost-free

5
How about a gesture interface?
6
Body tracking
Polhemus 2000
Teresa Martin 1997
7
Datagloves
8
Image hand tracking
Cullen Jennings, 1999
9
Our Approach

• Natural fingertip gestures

10
Whats natural

• Small - max displacement of 5 cm
• Gentle, finger snap)
• Few gestures, little memory work
• Avoid ring and pinky finger
• Examples
• Thumb as anvil - index, middle as hammer
• Thumbpad to fingerpad
• Thumbpad to fingernail edge

11
Fingertip Gestures

• Tap, double tap
• Finger and thumb pads rub
• Money gesture and reverse
• Finger and thumb pads press
• Soft Flick

12
Fingertip Gesture Interface

• Wristband-mounted piezo-electric contact
  microphones positioned on the styloid bones

• Sense bone conducted sounds produced by gentle
  fingertip gestures

13
Simple Classifier

• Allows real-time analysis and control
• 800 samples every 10th of a second
• Take max absolute, quantize to 10 levels
• Finite state machine outputs Taps and Rubs
• Intermediate states filter background noise
• Buffer states allow continuous gestures
• Surprisingly accurate 90

14
Example Signals
15
More Sophisticated Classifier

• Noticeable differences in audio signals
• Hidden Markov Models
• Gesture and noise models trained with sampled
  data
• Confidence levels for each trained gesture

16
HMM Classifier Accuracy

• Using 3 subjects, collected 100 instances of
  gestures rub, tap and flick
• 80 used for training, 20 for testing

17
Wrist Display Prototype

• Timex Internet Messenger watch
• Tap to cycle through messages
• Double-tap to rewind

18
Other Prototypes

• Cellphone dialing application
• Rub scrolls list in one direction
• Tap dials phone number
• Powerpoint slide control
• Tap moves forward one slide
• Double tap moves back

19
Future Work

• Miniaturization of device
• Hitachi SH5 controller
• Improved gesture classifiers
• Finger Identification
• Analyze signals from multiple microphone
  locations
• User Studies
• Usefulness Compare performance to current
  cellphone, PDA and desktop control interfaces.
• Social impact Study how users exploit private
  control techniques to mobile devices

20
Conclusion

• Fingertip gestures
• sensed acoustically at the wrist
• can be communicated wirelessly to nearby devices
• show promise as a control method.