Scriboli: High Performance Pen Interfaces

1
Scriboli High Performance Pen Interfaces
Ken Hinckley Patrick Baudisch Gonzalo
Ramos Francois Guimbretiere Microsoft Research
2
Scriboli Design Goals

• Speed without keyboard hotkeys
• Cognitive footprint diminishes with use
• Recall / declarative ? procedural skill
• Minimal demands on visual attention
• Repeatable motions for the user
• Predictable dependable system behavior
• Expressiveness
• Support many command structures apps
• Economy of design
• only a few things to learn

3
SKETCH Brown University

• Adding 1 more gesture breaks everything
• hard to learn, gestures specific to 3D editing

4
Recognize Pen Gestures?

• Classic problem ink or gesture ?
• Hard problem any gesture could be ink
• Inferred decision comes too late
• real-time interactive feedback difficult
• Make it a physical skill
• All we need to solve this is ONE BIT of info
• physical button on bezel
• habit-forming, deterministic, low attentional
  demand
• nonpreferred hand, no trip to lasso mode

5
Ink/Gesture study w/ U. Washington

• 5 Techniques
• Barrel Button
• Taphold
• NPH button
• Pen Pressure
• Pen Eraser

6
Phrase Structure for Scriboli
Scope
Command
Parameters
Separator (start)
Separator Scope/Command
Separator Command/Parameters
Separator (Done)
7
Phrase Structure for Scriboli
Scope
Command
Parameters
Separator (start) Press Gesture
Separator Scope/Command
Separator Command/Parameters
Separator (Done)
Fast and reliable Sellen 90, 92 Can be chunked
by users
8
Phrase Structure for Scriboli
Scope
Command
Parameters
Separator (start) Press Gesture
Separator Scope/Command Pigtail
Separator Command/Parameters
Separator (Done)
Fast, based on muscle memory Reliable and
easily chunked by users
9
Phrase Structure for Scriboli
Scope
Command
Parameters
Separator (start) Press Gesture
Separator Scope/Command Pigtail
Separator Command/Parameters Crossing boundary
Separator (Done)

• Merge command selection and direct manipulation
• FlowMenu Guimbretiere 00, Control menu Pook
  00

10
Phrase Structure for Scriboli
Scope
Command
Parameters
Separator (start) Press Gesture
Separator Scope/Command Pigtail
Separator Command/Parameters Crossing boundary
Separator (Done)
One stroke for scope, command, and
parameters Non-modal system (quasi-modal)
11
Delimiters for scope selection marking

• A delimiter is something different
• lexical structure of stroke
• e.g. self-crossing gesture stroke
• DEMO / VIDEO delimiter techniques

12
Results Completion Time
Button Handle Pigtail Pigtail2
Timeout

• Left bar main block right bar repeated block
• Dotted bars are Pigtail2 design iteration

13
Results Learning Effects

• For the 24 trials of Repeated Invocation Block

14
Results Error Rate
Button Handle Pigtail Pigtail2
Timeout

• Error rate of selecting wrong marking direction

15
Summary Grammar for pen input

• Scriboli proposes fundamental building blocks
• Links together object, verb, indirect object in
  fast, fluid, unambiguous command phrases

16
Stitching Pen Gestures that Span Multiple
Displays
Microsoft Research Ken Hinckley Gonzalo
Ramos Francois Guimbretiere Patrick Baudisch Marc
Smith
17
Wireless Network 2004 Horseless Carriage 1904?

• Are Wireless Networks really just wired networks
  without the wires?
• Or are they something completely different?

18
Wireless Device Soup Which links does the user
want?
19
Name That Device

• Users need techniques to intuitively form bridges
  between devices
• How do users name the devices to connect?
• What is type / purpose of the connection?
• Parameters? (Who copies what, to where?)

20
Stitching

• Pen stroke that spans displays
• Move the pen
• Cross over bezel
• Finish stroke on nearby tablet
• System infers connection

21
VIDEO

• AVI04
• ACM Advanced Visual Interfaces 2004

22
Establishing a Connection

• Recognizes timing dynamics of pen trajectory
• There is nothing special about the pen!
• Wireless signal strength determines nearby devices

23
Automatic Screen Calibration

• Fits a line to absolute coords of users pen
  stroke

24
How Do Users Share Physical Space?

• Sociology?Proxemics How people use space
• Invisible bubble that surrounds an individual

25
6. How Do Users Share Physical Space?

• Sociology?Proxemics How people use space
• Invisible bubble that surrounds an individual

• Homework assignment
• Sit right next to someone at airport (when it is
  not necessary to do so)
• Time how long it is before the other person
  leaves

26
5. How Do Users Share Physical Space?

• Sociology?Proxemics How people use space
• Invisible bubble that surrounds an individual

27
(No Transcript)
28
(No Transcript)
29
Proxemic Lessons for Stitching

• Dont require contact touching is taboo
• Do they have to be right next to one another?
  10-40cm
• wide screen would be nice for collaboration
• no two faces trying to peek at only one screen

30
Proxemic Lessons for Stitching

• Dont require contact
• Do they have to be right next to one another?
  10-40cm
• wide screen would be nice for collaboration
• no two faces trying to peek at only one screen
• Gives users flexibility to be involved, or not
  mood, task,
• Intimate Spaces Combining screens. For close
  collaboration with friend or trusted colleague
• Personal Spaces Tablets can be separated by up
  to 30 yet still possible to stitch to give files
  to colleague, etc.
• Social Spaces Once connected, transporter can
  be used to give files to a user beyond arms reach

31
Ongoing work

• Multiple Devices Cooperative Stitching

32
Ongoing work

• Multiple Devices Cooperative Stitching

33
Questions?
34
Handle vs. Pigtail

• Handle can get in the way
• Adds some visual clutter
• Must check to be sure landed on handle
• Pigtail handles more than one scope elegantly
• Self-referential gesture