Enabling%20Natural%20Interaction

1
Enabling Natural Interaction

• Randall Davis, Howard Shrobe
• Aaron Adler, Christine Alvarado, Mark Foltz,
  Tracy Hammond, Mike Oltmans,
• Metin Sezgin,Olga Veselova
• MIT Artificial Intelligence Lab

2
Consider This Device...
3
Natural Interaction?

• What if we had to deal with one another as we
  deal with our software?

4
Lets Talk
5
Our Vision
6
Our Model
7
Rationale Capture

• Rationale capture doesnt happen.
• Because its too much trouble
• Rationale capture tools dont work
• Because theyre too much trouble
• The key it needs to be less trouble than its
  worth.
• The key to less trouble break down the barriers

8
Sketch Understanding
9
Why Sketching?

• Informal
• Succinct
• Natural
• High in information content
• Preserve topological and spatial properties

10
Sketch Understanding

• Requires intelligence
• at the signal level early processing
• at the interpretation level handling ambiguity
• at the engineering level intended behavior

11
Other Research

• Additional domain software diagrams
• Multi-modal interaction
• Describing and learning new icons
• Generating efficient recognizers

12
Sketching Software
13
Sketching Software
14
Sketching Software
15
Sketching Software
16
Sketching Software
//Source file c\\video\\Deck.java public class
Deck extends Hand implements Dealable
public Card theCard public Game theGame
/ _at_roseuid 3C21348C0257 / public
Deck()
//Source file c\\video\\Game.java public class
Game public Graphics theGraphics public
Deck theDeck public Player thePlayer /
_at_roseuid 3C21348C02E3 / public
Game()
//Source file c\\video\\Dealable.java public
interface Dealable
17
Multi-Modal Interaction
Three identical, equally spaced pendulums

• Graphical vs verbal communication Whats best
  said, whats best sketched?

18
New Icons
Define AndGate line L1 L2 L3 arc A semi-circle
A1 orientation(A1, 180) vertical L3 parallel L1
L2 same-horiz-position L1 L2 connected A.p1
L3.p1 connected A.p2 L3.p2 meets L1.p2 L3 meets
L2.p2 L3

• A shape description language
• A drawing sequence description language

19
Sketch Understanding Architecture

• Mechanical
• Engineering
• devices
• simple compositions
• primitive objects
• geometry rectangles, curves

20
Sketch Understanding Architecture

• Mechanical
• Engineering
• devices
• simple compositions
• primitive objects springs, joints
• geometry rectangles, curves

21
Sketch Understanding Architecture

• Mechanical
• Engineering
• devices
• simple compositions ratchet, latch
• primitive objects springs, joints
• geometry rectangles, curves

22
Sketch Understanding Architecture

• Mechanical
• Engineering
• devices mechanisms
• simple compositions ratchet, latch
• primitive objects springs, joints
• geometry rectangles, curves

23
Sketch Understanding Architecture

• Mechanical Software
• Engineering Engineering
• devices mechanisms
• simple compositions ratchet, latch
• primitive objects springs, joints objects,
  methods
• geometry rectangles, curves

24
Sketch Understanding Architecture

• Mechanical Software
• Engineering Engineering
• devices mechanisms
• simple compositions ratchet, latch hierarchies
• primitive objects springs, joints objects,
  methods
• geometry rectangles, curves

25
Sketch Understanding Architecture

• Mechanical Software
• Engineering Engineering
• devices mechanisms modules
• simple compositions ratchet, latch hierarchies
• primitive objects springs, joints objects,
  methods
• geometry rectangles, curves

26
Design Rationale Capture

• Make it less trouble than its worth
• Break down the barriers eliminate interfaces
• Natural interaction is enabled by intelligence
• About sketching, gesturing
• About the domain
• About the task