Pen

1
Pen Gesture Input
2
Agenda

• Questions?
• Next week
• Pen input
• free-form
• soft keyboards
• recognition
• Gesture recognition

3
Before we begin

• If this is interesting, you might explore Spring
  2000 courses
• UI Software Design (MacIntyre)
• Mobile and Ubiquitous Computing (Mynatt
  Starner)
• Pattern Recognition (Bobick)
• Special Effects (Essa)
• Computer Vision (Starner, Essa)
• Computational Perception (Starner, Essa)

4
Pen input

• Free-form ink
• Soft keyboards
• Recognition systems
• generalize to gesture-based systems

5
Free-form ink

• ink as data
• humans can interpret
• time-stamping
• implicit object detection
• special-purpose domain objects

6
Free-form ink examples

• Ink-Audio integration
• Tivoli (Xerox PARC)
• Classroom 2000 (GT)
• Dynomite (FX-PAL)
• FiloChat (HP Labs)
• The Audio Notebook (MIT)
• Flatland (Xerox PARC) - augmented whiteboard

7
Soft Keyboards

• common on small mobile devices
• many varieties
• tapping interfaces
• simple gesture recognizers

8
Tapable, hunt-and-peck

• QWERTY vs. alphabetical
• Can you think of other options?

9
T9 (Tegic Communications)

• Alternative tapping interface
• phone layout plus dictionary
• mobile phone interface

10
Unistrokes

• Single stroke handwriting recognizer
• Xerox PARC -- PARC Tab

11
Graffiti

• Palm Pilot
• Very similar to Unistroke
• Physical design matters

12
Quickwrite (Perlin)

• Single stroke
• Memorize?

13
Cirrin (Mankoff)

• Word-level single stroke recognizer

14
Handwriting recognition

• Lots of resources
• see Web
• good commercial systems
• Two major techniques
• on-line
• off-line

15
Mixing modes of pen use

• Users want free-form and commands
• or commands vs. text
• How to switch between them?
• (1 mode) recognize which applies
• (2 modes) visible mode switch
• (1.5 modes) special pen action switches

16
Correcting recognition errors

• Really slows effective input
• word-prediction can prevent errors
• Various strategies
• repetition (erase and write again)
• n-best list
• other multiple alternative displays

17
Other interesting applications

• Signature verification
• Note-taking
• group (NotePals by Landay _at_ Berkeley)
• student (StuPad by Truong _at_ GT)
• meetings (Tivoli and other commercial)
• Sketching systems
• early storyboard support (SILK, Cocktail Napkin)

18
Gesture Recognition

• Tracking 3D hand-arm gestures
• fiber optic, e.g. dataglove
• magnetic trackor, e.g. Polhemus
• Perceptual user interfaces
• emerging area
• mainly computer vision researchers

19
Advantages

• Using Wizard of Oz techniques, we find
• Consistent gestures for certain commands
• Easy to learn
• Remembered over time
• DFAB interaction framework

20
Issues with Gesture Recognition

• Does a gesture clearly indicate
• functional informatin
• scope
• target information
• Variability

21
Non-obvious gestures

• Biometric signals
• smart floor
• affective computing

22
Have a good break