Pen

1
Pen Gesture Input
2
Agenda

• Questions?
• Poster session 2 moved
• Pen input
• free-form
• soft keyboards
• recognition
• Gesture recognition

3
Before we begin

• If this is interesting, you might explore Spring
  2000 courses
• UI Software Design (MacIntyre)
• Mobile and Ubiquitous Computing (Mynatt
  Starner)
• Pattern Recognition (Bobick)
• Computer Vision (Starner, Essa)
• Computational Perception (Starner, Essa)

4
Pen input

• Free-form ink
• Soft keyboards
• Recognition systems
• generalize to gesture-based systems

5
Free-form ink

• ink as data
• humans can interpret
• time-stamping
• implicit object detection
• special-purpose domain objects

6
Free-form ink examples

• Ink-Audio integration
• Tivoli (Xerox PARC)
• Classroom 2000 (GT)
• Dynomite (FX-PAL)
• FiloChat (HP Labs)
• The Audio Notebook (MIT)
• FlatLand (Xerox PARC)

7
Soft Keyboards

• common on small mobile devices
• many varieties
• tapping interfaces
• simple gesture recognizers

8
Tappable, hunt-and-peck

• QWERTY vs. alphabetical
• Can you think of other options?

9
T9 (Tegic Communications)

• Alternative tapping interface
• phone layout plus dictionary
• mobile phone interface

10
Quickwrite (Perlin)

• Unistroke recognizer

11
Cirrin (Mankoff)

• Word-level unistroke recognizer

12
Recognizing pen input

• Graffiti
• unistroke alphabet
• Other pen gesture recognizers
• for commands
• measure features of strokes
• usually no good for complex strokes

13
Handwriting recognition

• Lots of resources
• see Web
• good commercial systems
• Two major techniques
• on-line
• off-line

14
Mixing modes of pen use

• Users want free-form and commands
• or commands vs. text
• How to switch between them?
• (1 mode) recognize which applies
• (2 modes) visible mode switch
• (1.5 modes) special pen action switches

15
Correcting recognition errors

• Really slows effective input
• word-prediction can prevent errors
• Various strategies
• repetition (erase and write again)
• n-best list
• other multiple alternative displays

16
Other interesting applications

• Signature verification
• Note-taking
• group (NotePals by Landay _at_ Berkeley)
• student (StuPad by Truong _at_ GT)
• meetings (Tivoli and other commercial)
• Sketching systems
• early storyboard support (SILK, Cocktail Napkin)

17
Gesture Recognition

• Emerging area
• Perceptual User Interfaces
• mainly computer vision researchers

18
Advantages

• Using Wizard of Oz techniques, we find
• Consistent gestures for certain commands
• Easy to learn
• Remembered over time
• DFAB interaction framework

19
Issues with Gesture Recognition

• Does a gesture clearly indicate
• functional informatin
• scope
• target information
• Variability

20
Non-obvious gestures

• Biometric signals
• smart floor
• affective computing

21
Videos

• FlatLand (Mynatt PARC)
• MERL (Freeman)
• Digital Desk (Wellner)
• StuPad (Truong)
• American Sign Language (Starner)