People, Pens, and Computers

1
People, Pens, and Computers

• François Guimbretière
• HCIL, University of Maryland
• francois_at_cs.umd.edu

Ken Hinckley MSR kenh_at_microsoft.com
2
New devices, old tasks
Middle picture from Sellen et al.
3
People, Pens, and Tablet PC
The New Yorker
Illustration from Ken Hinckley presentation at
Stanford
4
Typical setting for todays interface

• Fixed stable environment, with a keyboard,
• Indirect interaction,
• High precision pointing system

5
Typical Tablet PC use

• Portable, unstable environment, without a
  keyboard
• Direct interaction,
• Low precision aiming

6
Pen based interface for Tablet PC

• Interface framework
• How to make it more pen-friendly?

• Document area
• How to create a more fluid interface?

7
Pen based interface for Tablet PC

• Interface framework
• How to make it more pen-friendly?

8
CrossY crossing-based GUIApitz Guimbretière
04

• Empirical foundations
• Use of strokes to cross target is more pen
  friendly
• Crossing is as efficient as point-and-click
  Accot Zhai, 2002
• The basic interactor
• How expressive is it?

9
CrossY video
10
Previous Work

• Theoretical basis
• Steering Law, Trajectory-Based Tasks Accot
  Zhai 97-02
• Limited scope examples
• Toggle Map Baudish 98
• Lotus Notes multiple e-mail selection
• Conceptual design
• Visual Instruments Winograd Guimbretière 98
• Overloading
• Gedrics Geißler 95

11
Command composition

• From stroke-by-stroke interaction
• Borders are used to validate/cancel

12
Command composition

• From stroke-by-stroke interaction
• Borders are used to validate/cancel
• To multi-command stroke

13
Scrolling
Line by line area
Page by page area
Absolute area
14
CrossY scrollbar

• Overloading simplify interactions
• Shorter distances to issue commands
• Not as much precision necessary

15
CrossY scrollbar

• Overloading simplify interactions
• Shorter distances to issue commands
• Not as much precision necessary
• Extending stroke for repeat
• No need to wait for a timeout

16
Cursor control

• Cross to jump to an absolute position

17
Cursor control

• Cross to jump to an absolute position

18
Cursor control

• Cross to jump to an absolute position
• Near drag for coarse adjustment

19
Cursor control

• Cross to jump to an absolute position
• Near drag for coarse adjustment

20
Cursor control

• Cross to jump to an absolute position
• Near drag for coarse adjustment
• Far drag for fine adjustment
• Similar to FineSlider Masui 95
• But one single stroke

21
Use of directionality

• Continuous find and replace

22
Use of directionality

• Continuous find and replace
• Reverse direction for undo

23
What we learned

• Very well received by users
• HCIL open house
• UIST
• Space requirements
• Similar to point-and-click
• Trade-off with command combination due to
  sloppiness
• Overloading vs. easy discovery
• Consistency helps with adoption
• Known in Windowing systems

24
Pen based interface for Tablet PC

• Document area
• How to create a more fluid interface?

25
Scriboli and Stitching

• Kens presentation

26
My desk
27
Affordances of paper documentsSellen 01

• Easy to navigate
• Two-handed interactions and tactile feedback
• Reading across more than one document at once
• Easy to annotate
• Directly on the document or on a nearby pad
• Well accepted during meetings
• Socially accepted conventions
• Very difficult to modify
• Printed documents are created and edited as
  digital documents
• Expensive to distribute and archive

28
Bridging the gap previous work

• Digital emulation
• FreeStyle system Wang 89
• MATE Hardock 93
• XLibris Schilit 98, Golovchinsky 02
• Tight coupling
• DigitalDesk Wellner 93, Ariel Mackay 95
• A-Book Mackay 02
• PaperLink Arai 97
• Intelligent Paper Dymetman 98
• Paper as input device
• Xax Johnson 93
• Anoto
• Paper PDA Heiner 99, Avrahami 01

29
CohabitationGuimbretière 03
30
Stroke capture

• Requirement
• Stroke coordinates on the page
• Page ID
• Large address space
• Possible technologies
• Anoto
• DataGlyphs Hecht 94
• MEMO pen Nabeshima 95

From Anoto documentation
31
System architecture
32
My Desk with PADD
33
PapierCraft command system for PADDLiao,
Guimbretiere and Hinckley 05
PADD notepad
PADD document
Anoto pen
34
PapierCraft using PADD as proxies
Copy
Paste
Synchronization
On your Tablet PC
35
PapierCraft

• Gesture/Ink
• Use the command button
• Scopes
• Command selection
• Marking menu
• Writing down an unambiguous prefix of a command
• Batch mode processing

36
Early feedback and future work

• Small scale evaluation
• Commands with written words were very popular
• Higher recognition rate needed for strokes only
  feedback
• Contextual information might provide enough
  feedback during pastes
• Future work
• More reliable recognition engine
• Pen-based feedback
• Haptic feedback, LED
• Real time processing streaming stroke

37
Conclusions

• Four pen-based interfaces
• CrossY explores crossing-based interfaces
• Scriboli explores high performance direct
  manipulation interfaces
• Stitching explores multi-devices interactions
• PapierCraft uses PADD as document proxies
• One goal
• Bring the ease of use of pen and paper to digital
  interfaces
• Future
• Seamless integration of both media

38
Acknowledgments

• Collaborators
• Jim Hollan
• Students
• Georg Apitz, Nicholas Chen (CrossY/ScribolY)
• Kevin Conroy, Dave Levin, Chunyuan Liao
  (PADD/ProofRite)
• Sponsors
• Microsoft
• HP
• Anoto, Logitech, Maxell
• Colleagues and friends
• Corinna Löckenhoff
• Samrat (Bobby) Bhattacharjee
• Ben Bederson